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0.0 INTRODUCTION 


A data flow computer is machine with architecture radically different from 
that of existing computers. It can perform computations simatanmbusly on many different 
parts of a program. Ai Wypte date: Bete: compe: hae.epne eileen procenetrs, (a2 can 
utilize all of them simultaneously, each executing a different instruction. 


To handle arrays and other date structures, a dats flow computer must have a 
data structure processing facility. and memory thet hese. similar: facility to perform many 
Operations concurrently. Such a date structure memery.is the subject of this thesis. — 


A data flow camputer owes its great speed to its ability to perform many 
Operations at once, even though each individual operation is no faster than:on @ conventions! 
computer. The same is true of the memory. The mamery to be presented here hes a 
retrieval delay just es greet 26 conventional memories, singe no new circuit technology will be 
proposed. However, it hes en enormous data transior rate because-cf ite ability to handle ~ 
concurrent transactions. This concurrency is made. possible by an unusual type of interface 
celled packet communication. 


Section 1 of this thesis is an overview of data flow computers and the type of 
memory that such @ computer requires for structure proceseing. Sestion 2 ts a treatment of 
packet communication systems, showing how their behevior is defined. In section 3 the basic 
memory unit is described, slong with a “cache” mechanism and an “interleaving” method: to 
improve its performance. In section 4 an implementation of the memory using shift registers 
or magnetic disks will be given, showing how the. disedventeges of such devices can be 
Overcome through the use of packet communication.. Section.5 examines some sepects of the 
processing unit that uses the memory, and section 6 exemines: the. “deedieck” problem end the 
cost of overcoming it. Section 7 presents suggestions fer, future-rusearch.. 


1.0 DATA FLOW COMPUTERS 


As the need increases for ever faster computers, one technique for improving 
performance that hes drawn considerable. interest in the fest few years is @ radically new 
design known os o dete flow computer [6] (7} [11} 118]. A conventions! computer has only 
one locus of centzol, thet ie, ene point in the program at eny given instant at which 
instructions are executes: Ability te execute mare than Ont inatruction st 2 time can improve 
performance significantly, and some computers use en instruction lookshead to achieve this [3] 
[9]. However, the senetite ef lookahead methede:ere Vmited, end such computers ere 
enormously compiler. Other ettempts to incresee inetructied ceneurrency include “array 
processors” {16}, but such machines ore inflexible anal extremely difficult to program. . 


A dete flew computer achieves executional concurrency by using a different 
internal representation of the source program. Ineleed of repretenting the program as 2 list 
of ingtructions te be ewaculed in  perliculer order, the pregrom is represented as s date flow | 
schema, A date tiew scheme is 3 directed graph whose finded represent instructions snd 
_ whose ares sheve the date dependence’ smeng instructions. The eriler of instruction execution 
is determined solely by the dete dependence - an instruction te executed when elf of its dete 
sources heve produced results end all of ils destinations are reeily to receive dats. This 
allows meny instructions ee the program te a ee 


The dete in o dete How program can be madoled by “Whane” thet reside on the 
arcs of the graph. Esch orc mey cantein af moet one tober. “The execution rule for most 
instructions is as fallews: 


An inetruction (ether then « merge or gota) ie reedy for exeention whenever all 
of its input arcs contain tokens and oll of ite culput arce are empty. When an 
inatruction is executed; the tokens’ on the, input arcs are absorbed. The 
function denoted by the instruction le computed, uding ‘the values in the 
sbsorbed tokens se input date. A token containing the function velue is pleced 
on each output arc. . | | 
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There are a number of ways of handling decisions and iteration control. 
Perhaps the simplest is the use of special instructions M, T, and F. These receive e boolean 
value on one input (the “control” ey eee ee eee eee eee another 
Input, Their execution rules ere os follows: 


The M (merge) has « control input and two deta inputs labelled "T° and “F". To | 
‘be ready for execution, there must be a boolean token on the arc leading to its 
control input. Furthermore, the arc feading to whichever. of its: T or F input 
matches that boolean token. must have @ token, and elf eutput arcs rust be 
empty. When it is executed, the control token and the data teken st the input 
indicated by the control token are absorbed. Copies of the token at the 
selected data input are placed on each output arc. Input tokens ere not 
required at the non-selected data input, end if any are present they ere not — 


The T (true gate) and F (false gete) instructions have a control input and a data 
input. They ere ready for ‘execution whenever both input orcs contain tokens 
and all output arcs are empty. When they are sxecuted, the Inputs are 
absorbed. If the control input matches the neme of the instruction, copies of 
the data input ‘are pleced on the output arcs. If not, no tokens ‘ere placed on 


the output arcs. 


Constants can be generated through the use of funesjone, of no erguments. An 
instruction to perform such a function has no input arcs, so, in accordance with the execution 
rule, it places tokens on its output arc as fest as they are removed... 


Here is an exemple of a dete flow echame te compute the factorial function: 


Boolean inputs to M, T, and F instructions ere drawn as open arrows. Tokens existing in the 
initial configuration of the pregram ere drawn as fifled-in circies. 


The behevior of a dete flow scheme under the execution rules has @ very 
importent property - It is determinate. This moans that the output of the program is 
determined only by the input, and le Independent of the timing of instruction executions. All 
rune of such © program with the same dete wil yield the seme resus Determinacy follows 
from the facts that 


(1) Each instruction produces a result which is @ function only of the values of 
te Input toners, thet le, ech de ofthe echome lo delerminete 


(2) The value of a token does not chenge in any way white it resides on an arc. 


(3) The execution rules, end fact (2) shove, quetify the schema as a valid 
Interconnection of autonomous communicating systems. 


es 


It is an established result that such an Interconnaction of. determinate: systems is determinate 
(1) (14). 


1.0.1 DATA a COMPUTER ARCHITECTURE 


‘The memory system end: structure processor that ere the subject of this thesis 
are intended to be pert of = computer of the type described by Dennis and Misunes (6) [7]. 
Such a computer is composed of units which use packet communicetion [8] for transfer of 
.data. The only means of deta transmission among these units ls the. ‘trenemiasion of fixed size 
messages catled packets. “There is no clock or synchronizing information. 


The four main parts of the data flow computer ore . the tostrystion memory, 
arbitration ne network, func tional units, and distribution network, For itructure processing, the 
struc controller rand structure memory ere added. 


distribution ue arbitration 


To execute date flow program, its schema is encoded into the instruction 
memory. Each cell of the memory contains one instruction of the schema. At the time the 
program is loaded, each call is filled with the operation code (arithmetic operation, merge, 
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structure operation, etc.) and the address of its destinations. The letter are the cells to 
which outgoing arcs point. The instruction celts also have receiver registers to contain 
Incoming “tokens”. When all necessary receiver registers became full, an instruction cell emits 
an operation packet, conslating of its operstion cade, the date trem the receiver registers, end 
the destination addresses. | 


Any given program hes s greet number of instruction celts, each sending 
operation packets enly occasionally. These streame of pechety are merged by the arbitration 
network into 2 smelt number of danse streams. The pechets coming oul of the arbitration 
network ere sorted eccerding to eperation code and sent te the appropriate functional units. 
In the case of structure processing instructions, they ere sent to the structure controller. 
The functional units er structure centrelier perform the indicated eperetion and form, for each 
destination, # rueuft packet consisting. of the destination address end s copy of the actual 
result. The result packets go to the distribution network, where thay are sorted by address 
and sent to the epprepriste receiver ragister of the epprepriste instruction cell. (The 
destination eddrece includee the receiver number.) If the instruction ia 9 structure operation, 
Se ees es eel ee 

result packets beck during the course of its computation = 


The preceding description dons net quite implement the execution rule: An 
instruction coll should walt until ite “output orca’, thet-ie, the receivers of ite destinations, are 
empty before issuing an operation packet. There is ne wey fer on instruction cell to "see" its 
destinations’ receivers. The problem is remediad by using, where necessary, acknowledgment 
tokene sent from @ cell's destinations to the coll lent, The echnewledges are treated like 
invisible arguments, except thet they contain no date.When # cell is executed, it may send 
result packets to some destinations and echnowiedges to others. A cell is not ready to be 
executed until it has received all necessary reel axqgaments and ali necessery acknowledges. 
Acknowledges ere pleced in the program where necessary te ensure that, when a cell has 
received all arguments and acknowledges, its destinations’ receiver registers will be empty. 
These acknowledges should not be centused with the pechet acknowledges to be developed 
tater. _ , 


il 


A constant need not be implemented as a separate node of the date flow 
schema. It can simply be loaded.into the.receiver register of the instruction cell that uses it, 
and marked in such a way that the inetruction cell knows thet.thet register is always full, 


An additional part of the dete flow computer, not-shown in the preceding 
diagram, is the host computer, This is a computer of conventional design, which has access. to 
the memory units. end control. functions of the deta flew computer. It is used for diagnostic 
testing and for initial loading of the instruction memory end strusture memory. It. does not 
participate in the ectual data flow computation. 


1.1 DATA STRUCTURES 


in order te handle arreye end date shructures ina date flow computer, it is in 
most cases nececsery to allow single tehens to have erifire structures as their values. (Some 
progreme which use erraye of fixed size, such as Fourier trensforme and other signal 
processing algorithms, con-make de willy erreys of instructions with one token on each arc. 
However, this approach is tmpractical for very terge arrays or fer dynemic structures.) For 
The simplest type of structure thet permits full generality is the tinery tree, which is 
recursively defined: @ binery tree le on clemedtery “objiel” from some set, or is a 
concetenation ef twe binary trees. Such trese form the basis for the programming language 
LISP. [4] [13] For defiditeness, the structures used in « date few computer will be assumed 
to be binary trees. 


The “elementary objects” are oli dete values other then structures that the 
computer can hendie, plus the epecial object nil. Deere See en en ener 
integers, beolean veluss, reats, etc, 


The principe! operation on e date structure is selection. A simple selection 
tekes a structure and 6 single bit. If the structure ie elementary end net nil, the result of the 
selection is undefined. If the structure is nil, the result is nil. Otherwiee, the structure is the 
concetenstion of twe structures, and the result of the ealection is the first or second of these 
if the bit is zero or one reepectively. A compound selection tehes a structure and a string of 
bits, end gives the result of applying simple selections repestedty, using the bits in sequence. 
The bit string is celled the selector. Let $ be the following structure: 


aie 
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SELECT[S, ‘1"]) =5 (a simple selection) 
SELECT[S, 001°] = SELECT[SELECT[SELECT[S, ‘0°], 0°] ‘1"] = 4 (a compound selection) 
The true “meaning” or “value” of a structure can be defined to be the set of 


ordered pairs of selectors thet yield elementary values other then nil, along with those values. 
Thus the structure S denotes the set 


{ $000", 1>, <001’, 4>, <O11'; 3.14, <1’, 5> } 
Nil simply denotes a substructure with no elementary items at ail. 

Using this definition of the meaning of a structure, there is a structure 
corresponding to any finite set of ordered pairs of selectors and aiastentery values (excluding 
nil) such that no selector in the set is an initial substring of enother. The structure nil 
denotes the empty set. . 

SELECT[struc, sel] = 
The elementary value v if struc conteine the:-peir <est,:v> 


Undefined if <s, v>.« struc. where ¢ is-e propec initial eubstring of cel 
The structure { <s, v>-|-<selea,.v> « struc } otherwise: 
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Structures cen be built with the append cperation. APPEND places @ given abject (structure 
or elementery value) ente @ given structure with @ given selector, removing whatever 
substructure previeusty euleted there, In the sat-theoretic medal, 


APPEND{ctruc, new-vel, sel} = 
(struc - { <a, v> | one of sel ore is sn initiot aubetring of the other} U { <sel, new-val> } 
if new-val is elementary. 
(strue ~ { <a, v> | one of anf oF ip an initial eubstring of the other} U 
{ <col-e, v> |<, v> @ new-vel] ¢ new-vel ie a eteustura, including pil. 


Letting S be the structure defined previausly, APPENEES, 7, ‘01") is 


The substructure containing nil and 2.14 disappears. 
1.1.1 REPRESENTATION Ht MEMORY 


Structure can be implemented en @ date fiew computer in the same way that 
they are commanty hmplemented on ordinery computers - a0 linked llets of “cells” in a memory. 
An elementary object is represented by the object itself. A concatenstion is represented by 
the address in memory af ¢ cell conteining the represantations of the twe substructures. In 
either case, e structure ie represented by & email enewil ef infermation. The huge amount of 
information thet conetitutes the structure Heelf Hea ineide tha memory, and the representation 
le merely a pointer te this. The eperation of selection le quite simple. Calls ere reed from 
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memory and the eppropriate halves of the date used, under contro! of the selection bits. 
1.1.2 SHARING 


Such an implementation leads to the possibility of a single structure in memory 
being shared (or partly shered) by several parts of the computation. In a data flow computer, 
two tohow might: have the come. polnter 26 Ttr valve ‘This feof course very desirable for 
modification of pointers inside the memory:can chenge.tbe-welue-of structures. other than the 
intended one, if structures have perts in common. Ip.msny programming languages, ‘this Is 
considered a reasonable and even desirable effect. For example, the LISP language has 
instructions to modify existing structures. In a data flow computer, however, this cannot be 
permitted for reasons of determinacy. In order for a data flow computer to be determinate, 
the meaning (in the set-theoretic sense given previously) of a token bearing a structure vaiue 
must not change while that token resides on an arc. Since other instructions, including 
APPEND’s, can be executed while a token resides on an arc, APPEND must never change any 
substructures that are shared with other structures. 


In the proposed structure processing facility, each cell has a reference count 
which makes it easy to tell whet substructures are shered. Whenever the APPEND processor 
is tempted to modify a cell that is shered with another structure, it makes a copy of the cell 
and modifies the copy insteed. For example, if S is a pointer to the following structure in 
memory: 


where the number in each node is the reference count, APPEND{S, 7, 01") yields - 
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1 4 nil = 3.14 


The node that originally had a reference count of two may not be modified, so a copy is made, 
and its reference count is therefore reduced to one. The structure controller to be described 


in the next section will perform these tasks. 
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1.2 THE STRUCTURE CONTROLLER 


In this section we will outline the behavior of a processing mechanism that uses 
the structure memory to provide a structure facility for the dete flow computer. The basic 
behavior of the structure controller is that it receives operation packets from the arbitration 
network and delivers reeult packets to the distribution network. It holds the state information 
for structure operations in progress, and performs memory operations by sending packets to 
the memory and receiving packets in return. 


The purpose of this section is to show how the structure controller will use the 

memory, rather than to give a detailed specification for the structure controller. Therefore, a 
number of design decisions will be made arbitrarily, For the most part, the requirements of 
the structure memory are independent of these decisions. For example, the memory design 
would not change if ternary trees were used instead of binary ones. 


Some espects of the design of the structure controller will be considered in 
more detail in section 5. 


1.2.1 DATA FORMAT 


The memory space is divided into “words” or. “calls", each of which holds one 
‘node of a structure. Since the memory Is used for the storage of binary trees, the words 
representing nonterminel nodes contain twa, peinters to ether nodes. The convention will be 
made that all words of the memory will be divided into helves, called the left haif and the right 
half. Each half has an “elem” bit bit indicates whether it conteins an slementery item (terminal 


node) or a pointer to another word in the mamory. If the bit.is-1, the half word contains an: 


elementary value. The interpretation of thet half word.is then. the exclusive responsibility of 
the rest of the computer, unless it Is wil, The structure controller treate any slementery value 
other than nil simply es a collection of-bits. Any type information (integer, fleating point 
number, character, etc.) must be encoded into the helf word siong with the date. 


The structure graphically represented as follows: 


ees | es 2 


(& different convention could be: used, in: whith: cack elementary value takes an. 
entire: word nctued:ef halb'e- werk The two conventions ere-equally powerful, end differ only 


Ah words: of memary thet are: not part et & étructore: ere: kept ine collection of 
free storage liste. (There eve severed cocty Nets, rattior then one, in teder to maintain a high 
rate:.ef peocnasing. This: point will be diecussed in section 805.) Whenever the structure 
controller needs « word inv: order te: ereate & node, it tukes: it frow ofa of the fists. Whenever 
a node je destroyed; tit te, alt peivters te it dikappesr; the werd contsining It is returned to a | 
free storage list. =D oe : 
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Each node of a structure has a reterence count, which is the number of 
pointers to that node thet exist, whether in other nodes. or in the rest of the computer. (The 
latter includes operands waiting in instruction celle-and packets in transit through the 
arbitration and distribution networks.) The structure controller incresses or decrseses the 
reference count of each node as pointers to it are crested and destroyed. When the 
reference count is decreased to zero, the node disappears, :se::it ie: returned.to « free storage 
list. Winenever. this. heppeces Any -peintecs. that eames sensi’ seeppeer, and so the 
reterence counts of the nodes pointed to must be decreased. 


The choice of 2 reference count strategy for memory. management instead of 
the “mark and scan” method commonly used in LISP. syateme wes made-for three reasons: 


(1) The mark and scen method requires a garbage collection operation which 
must find every reference to every structure, Since references exist in 
packets in transit, it would be necessary, to stop the.entire- computetion end 
wait until ell packets stop moving. ‘before. io aaa commences. 


(2) The reference coynt is needed anyway in. order.to. implement.the copying 
rule efficiently. Whenever the structure. costralier needs: to:modity a node 
es part of an APPEND operation, it may do.sa sefely if the reference count 

_|s one. If not, the node must be copied. 


(3) The ‘objections to the reference count methad in many list processing 
systems, that it is difficult to recover circular iste, dows not apply here. 
Because of the copy rule, circular lists ere never created. . 


1.2.3 THE STRUCTURE OPERATIONS 


The structure controller to be proposed implements the following program level 
operations: 


SELEC Tistructure, selector) - The selector ie a bit string of definite length. The 
structure ie: tesed: under contre! of the bits. in the selector, starting with 
the taftremet bik: A. mesa bit enlunts thie telt oftngritig and. 2 ene bit selects 
the right: The item-et the-satected paket lit the: structure ia returned, 
whathar it is. elementary or substructure. 7 | 


APPENOtateucture,. ehject, selector) ~ feturna @ structure similar to the given 
one, but having the object st the place speciting bythe selector. Whatever 
was et thet place: in the eriginel etructere te sheent in the result. The 

’ object may be elementery or a structure. Any pert of the original structure 
that ie: -ahered: will: ther parte of the computation le not medified. The 
controller coplee part er ait of the origina) etructure. os necessary to be 
sure thet this in the case. 


The structure contrelier ‘reeogninns the special conatent nil which, while 
elementary, is also. the structure with no selectors: Mit is used a2 2 terminal node of a 
structure to indicele: that there are: ne: objects theyend thet point. Any pert of a structure 
may be deleted simply ly using the APPEND cperstion ta replace it with nil, and a structure 
may be cresbad by appending aomething: 40 nik. tis aneumad thet the constant nil is explicitly 
available: te: the peegrenmer: ter thats jitirpiene, The tantidllie optimizes aii structures, 


There: are two. more: eperetions performed implicitly by the controller. If any 
ct the result must be: anprspietately increcced. Alen, if any apération discards @ structure 
value, the reference count must be decreséed: ‘tt follows that the conditions! operations such 
as true and felee enters must be executed hy # structure centrolier it the objects being 
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1.2.4 THE MEMORY OPERATIONS 


The structure controller communicates with the memory by sending command 
packets and receiving result packets. These packets are given names describing the 
Operation to be performed. 


To read « word of memory, a FET ("fetch") packet is sent, giving. the address. 
The memory returns a LOAD packet with the date, Gefwaen. the FET and the corresponding 
LOAD, many other packets might be sent and received. This is a consequence of the 
paralielism of the data flow computer: just a¢ with the other functional units, the rate at 
which structure operations are performed can be increased by silowing many operations to 
be in progress simultaneously. This concurrency is mede possible by the use of packet 
communication at the memory interface. The FET packet thet begins.en operation and the 
LOAD packet that ends it are distinct events and might be separated by a great number of 
other packet transmissions and receptions. Each LOAD packet Is identified with. the FET 
packet that caused it by means of the "tag", to be described later. 


Each LOAD packet conteing the eddress of the word and its reference count, as 
well as the data. The address is probably not used by the. strugture. controller, bul is included 
as part of the specification of the memory module because. it is needed.by the cache 
mechanism to be described in section 3.2. The structure controller uses. the reference count 
in order to tell when « node may be written on without being copied (if count = 1) and when 
a node should be destroyed (if count = 0). 


To increase or decrease the reference count of @ word, the FET* or FET” 
packets, respectively, ere sent. These are similar to FET, except that the reference count is 
first modified. The memory replies to-them with LOAD* or LOAD” packets which are similar to 
LOAD packets. In some cases the structure controlier does not use the data in » LOAD* or 
LOAD” packet, but it does not really cost anything for the memory to send it. 


To write on a word of memory, the structure controller sends an UPD 
("update") packet giving the address, data, and reference count. The reference count is 


presumably one, but Whe epecificution wf the memory module stews af arbitrary count to be 
given. (In an actual implementation of a structure controtier endl memory, unnecessary fields 
would be omitted where pevsiite, oe thet the controler would rat tend @ reference count in 
UPO peenete or recetve an address in LOAD, LUAD®, er LOAD pachete.) The memory sends no 
repty te an UFO pectat. 


There % anether command that the memory revegnizes. The CLR packet waits 
until af pending cperstions on the given word are complete, end then returns @ DONE packet. 
It ie rot weed Gy fhe etructere contreter at of, But 6 required for operation of the cache. 


1.2.5 THE TAG FIELD 


Every FET, FUT’, or FET” packet Nes @ Hold cafted the “tag” fieid thet 
constitutes o roninder trem the etrusters comtrelter 10 Hoult, teling ft what to do with the 
result of the operation: ‘Tro tay field ot © command pedhet fe returned unchanged in the 
result packet. 


Consider the cease of 9 simple SELECT wutruttion. When the instruction cell 
_ fires, on eperatien peetat gous te the eirecture contreder centeining the operation code, the 
structure, the selector, and the-sddrenees of the the ietruction culls whieh are to receive the 
result, There wight typtoally te three such deutmetion eddrevees, each wbout 20 bits tong. 
The structure eontrelier con ehnply mate thom tHe tag Held of the “fetch” command to the 
memory, and then wee them when they come beck in the result pachet. In the case of more 
complicated structure operations, such as APPEND’: with compound selectors, there is « lerge 
amount of state information thet must be remembered through the many memory transections 
that moke up the strustere operation, In addition te the destination eddresses, there is the 
detum to be appended, the structure te be uitimetely reterned, the remaining selector bits, 
end a few pointers, The totet enount of such date typically might be 200 bits or more. 


There are two ways of handing fie information. One method is to include ell 
of it in the tag field of commends to the memory, so the structure controtier duesn't need to 
store ény information sbout the stete of ongoing structure operations. When the result 
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packet comes back from the memory, the structure controller looks at the entire packet 
including the tag field, decides what to do. next, end. produces, a.new.gachet.teo send back to 
the memory. This method (the "memoryless structure cpptrotiec”: methed).is efficient, bat it 
requires en ‘extremely wide date path for all memory trenspetions,.ond.it gives.cise:to very - 
difficult problems of avoiding deadiacks. 


A second method is to store all of the state information in the structure 


controlier. This requires that the controller have a memory will capacity of 200 bits or 


more for every structure operation thet can be in progress at one time. In this case only the 
address of the block of memory.in which the stele information je: alered:must:be:-put: in the tag 
field. If 256 simutteneous structure operstions ere sliawed, the tag: Held only needs tobe 8 
bits. | 


In either cease, commends to the memory contein a tag:field. The memory 
echoes the tag back to the controller in the result packet, 


1.2.6 THE DATA AND REFERENCE COUNT FIELDS 


The contents of each memory word consists. of 9 date fleld and @ reference 
count field. The deta field is further divided into. two. pointer: falda, Jeet nate indicator bits, 
perhaps a bit to indicate thet the cell is on the free. storage list; and porheps.type- indicator 
fields for elementary values. All of these are significant only to the structure controller, and 
are irrelevant to the memory. The memory cen simply consider the seta te -be:2 swmogeneous 
field. In practice, it might be about 40 to £0 bits long. 


From the memory’s standpoint, the reference. count is.simply part ofthe data . 


associated with each word. In come transient cases it might become negative in.seme parts of ao 


the memory system, although the structure controtier will _npver,see a nagative reference 
count. In a typical realization, the reference count fleid might.be.sbout &.$0.15 bits-long. 


Incoming and outgoing peckets that read or write a word of memory have data 
and reference count fields that correspond precisely to the fields in memory. 
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There is a partial order on histories: X < Y if X is en initial subsequence of Y. 
For exemple: 
(15334) $ (1335437) 


but (13 25 4) and (15 33 4) do not satisty this aston not order. 


Since histories only grow Venger be time (progresses. and ‘inbele already in a 
history never change, a history et a later instant is always greater than or equal to a history 
at an earlier inetent. 


The length of port history X is denoted [X|. The individual peckets of X are 


My eXpe es Xpqe 


There is no defined time order ameng pastel arrivale cndibneent ports, so.it is 
useless to represent them es a single sequence. Instead, a history array is used, which is a 
collection of histories, one per port. The partial order on histories:cen.be extended to arrays: 


A 2 Bif each history of A is greater than or equal to the corresponding history of B. like. 


histories, history arrays increase as time progresses. Pe an eeeae 


The description of how a system is expected to behave is quite simple. It is a 
description, for every input history array, of what output hislery-errey the system will 
eventually produce. “Eventually” means in finite time for finite histories. For infinite 
histories, it means thet, for any K, the first K packets. will be produced in finite time. This is 
because a system which is expected to have an.infinite output. history cannot ever transmit its 
entire output in finite time. 


A description of the dependence of output history arrays on input arrays is 
called a functional specification. It isa description of how a.gystem: is expected to behave. 
The major probleme in the field of packet communication systems are proving that a system 
built in a certain way obeys a cortain functional specification, and proving that the 
interconnection of systems known to obey certain functional specifications obeys some other 


functional epecification. 


At, tor any ‘pdt array, the functional eperttication etetes ‘that there is only one 
possible output erray, the system is determingte (sometines called ‘functional, but that term 
will not be used hove). In that vase Swe ds 2 duneétivn, sey 4, mapping input arrays to output 
arrays, such thet, if inpot X fend we were) ts abdliversd te the eystem, 4X) will eventually be 
produced. If tarther inpdt tt thei’ given, ‘The input ‘Netery 4c Y with ¥ > X, end output history 
£01) will be prodsced, Sheen the eystom carat rotract amy a tts gravious output, f(Y) > 1X) 
Frew tia te ony tasers Ge erst Wt: 


K2>VwKhY >) 


‘IE thave 4 more than one taysl Tesyorme to 2 given input erray, the system is 
nondeterminate. in thet cave o tanction.ts le ised Se tistine tee Tendhionsl specification, but 
0X) fe the spt of oli inyal cutpat tistory ervaye. Functions aiafining ‘the ‘specifications of 
nondaterminete ayelums deb ‘Ubey «worl Ut weretenidity jprepelty, wliich will be given later. 


R te pousioks for wn intercomecten of nondelerminate systems to be 
determinate. ew ee ts arbitration 


\oudion in os espn ati eae tab sendin tian pot < 
2.0.2 DESCRIPTIVE ‘SPECIFIONFIONS 


‘Sinon a ‘major tack ‘of the ayetom aevigner is ‘to-damondirate ‘that a system built 
in u certain way Sbeys weriuin fonctions xpucitications, it 4 neewecery to describe in a 
ressonebly forme! way ‘now «system te built. A wiring diagram is one ‘tormatiem, but it is far 
too rigid and inplomentation-dependent. A higher devel muthed +s newtiod. When 2 system is 
sesombted trom components, ai using the packet communiottion principle, if is of course easy 
to describe the ‘iiterconmaction, ‘tulting what ports of the various eyvieme itre connected to 
each otter. For ydtone ‘elt Ternet tbe wb decomposed, the deacriptive epecttication will be 
given in terwe of = prayrem written in on vitrendly | ‘irformel RUBOL-llke lenguage. This 
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language is @ subset of the Architecture Description Language [10] which is under 
development. 


In the language we will use for giving descriptive specifications, packets will 
look like date records with a title and one or more, data fields, for example: “WRITE(3, 7). 
This format is purely cosmetic. In the actual hardware implementation, a packet is nothing but 
a collection of bits. The fields are simply divisions of these bits into subsets thet the sender 
and receiver both agree upon. The titles are just encodings of another field. 


2.0.3 AN EXAMPLE OF A DETERMINATE MEMORY 


A functional end descriptive specification of a system called MEM will now be 
given. MEM Is a random access. memory. with en input port IN end.an-output port OUT. Two 
types of packets may be delivered to it: _ 


WRITE(addr, data) writes the data into the given address 
READ(eddr) fetches the date from the given address. 


The “addr” and “data” fields contain numbers thet range over some finite and fixed spaces. 
There is one output packet type: 


RTR(addr, ’ data) - 
(RTR stands for "retrieve") 


Every READ packet delivered to MEM reeults in transmission of @ ATR packet 
bearing the address and the current contents of the memory. Every. WRITE packet stores its 
data in the memory. and returns no result packet. The initial contents.of each address of the 
memory is zero. 


For a given input history, the contents of the memory may be easily 
determined. The contents of each word Is simply the date field of the lest WRITE packet | 
having that address, or zero if there is no such pecket. The function fen realized by this 


memory is: 


ten 
it X = input Nictory and Y = cutpat history, 
tay « ¥ whore 
eee Se 
RTReddr, deta) if the i* READ(--) in X is READ addr) 


and The Sout WRITERIGNE «3 45 1 belore Wat READ 
40 WRITGleddr, data}, # there is euch « WRITE 


oe et Tt ie eT Ty 
tout Here is no WHITEleddr,~-) before 2 


Notation: ee en re eter field snd 
anything et ait in the dete Held: - ee 


fey» thet is, thet is the input History X is prevented te it, it will everitusity transmit output 


This specification says nothing explicit sboul the states of MEM. This is a basic 
property of the history function approach te system epecification - even for a device whose 
purpose is to hove etalon, uch ass memory, the specllieation dees tat mention the states. Of 
course, the memory dew have states, atid the state iss fonction of the input history. Since 
the inpet history reverede:wi-of the iteration thet Hes ever gone ib the system, it contains 
enough information te determine the stele. 


We new whew how the system MEM wey be Oult: The system uses a real 
random access qemery; with @ Capacity of one werd fer euch poutivie veiue of the “addr” 


ea, Sat 
yee STs 


field of incoming packets. We choose some obvious correspondence between the values of 
the “addr” fleld end word addresses. Each word can contain any of the possible values of the 
“data” field of incoming WRITE packets. We choose some obvious correspormience here also. 
The memory is initialized with ail words containing zero. 


The algorithm of the implementation of MEM is as foliows: If a packet ‘ 
WRITE(addr, data) is received, the date field is written.into mempry-at the: word:eddress given 
by the addr field. If « packet READ(eddr) is received, the.word:at the!sppropriate address: is 
nondestructively read, and s packet RTR(addr, data) containing the data fetched from memory, 
is returned. 


This eystem mey be implemented by the program which follows “Memory” is 
an array which represents the. actual memory. 


process starts at A 
inout port IN, 

var command, addr, data 
erray memory init 0 


| weit for input 


A: _—_ until packet Is available at IN do; 
command := packet from port INN 


| analyze input packet 


if command = READ(--) then 
let command = READ(eddr);. 
send RTR(addr, memoryleddr)) at port OUT oe 


K 
let command = WRITE(eddr, dete 
memerytaddr) :~ data; 

pate A 
Notes: 


(1) The statements for receiving end tranemitting pechets are excessively primitive. Slightly 
improved versions will be presented later, 


(2) The expression RTRieddr,deta) means “a RTR packet wheee fields are filled with the 
current values conteined in addr and dete’. 


(3) The "--" in conditionals hee its usual meaning. “If packet © WRITELS,--)" means "if packet 
is a WRITE packat whose first field ie 3°. 


(4) The “let packet = pattern” statement is on sesignment statement thet sets the variables 
appearing in the pettern to have the values of the corresponding fields of the packet. “let 
thing = WRITE(eddr,--)" means “if the type ef thing ia not WRITE, it is an error; otherwise 
set addr to the first field of thing end ignore the eecend field”, 


We now srove thet this implementation setisties the specification f.., . First, we need to 
show that the memory stele equels the system state (as defined by the input history) under 
the following correspondence: 


For all X, the contents of memory address X for e given Input Wistory Is 


zero if the input history contains ne packets WRITE(X,--) 
Y if the histery does contain euch packets, and the lect ie WRITEQ.Y) 


Proof by induction on the length of the history at port IN. For length zero, all celts contain 
zero by initializationxend the histery conteins ne WRITE packets af sil. Otherwise assume 
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true for any history of length K and prove it for K+1. 


If IN,,, = READ(--), nothing was written into. memory between receipt of IN, 


and IN,., , 50 the memory state did not change. The existence of WRIZE(--,--) packets did 
not change either. 


If INy,, * WRITE(addr, date), no memory cell other then. addr changed, and the 
existence of WRITE(X,--) packets did not change for X = addr, The contents of memory cell 
addr is now dats, and the last WRITE(addr,--) in the history. is now obviously WRITE(eddr, 
date). " 


Next, we prove correctness of the implementation. If the input history = X, we 
will show that f,..(X) will eppeer at the output. This proof is also by induction. If | = 0, 
fueu * But the implementation specifies no output except in response to input. Now 
suppose X” = KiKo nm Xyhy, - Lot x mK Me Xy- By induction, fueui™) appeared at the 
Output when X was the input history. When xq, arrived, the system transmitted no output if 
Xy., Was @ WRITE, and transmitted RTR(addr, memory(addr)) if x, wes READ(addr). 
Therefore the response to X’ is 


fey) concatensted with 
€ if xy, = WRITE(--,--) 


RTR(addr, memory(addr)) If xy,, = READaddr), where the memory 
state is that left by X 


Now Hey OH = Higy(XN + 1 if xy, is READ(--), which is the length of the 
response to x’. 


Also, if Xy,, = WRITE(—,--), Figg) = feu (X), and if xy) = READ(addr), f peu (°) 
= fuseyi%) concatenated with RTR(addr, z), where z = the data field of the last WRITE(addr,--) 
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2.1 NONOETERMINACY 


Nondeterminate systems can teke a wide variety of forms, and the problem of 
formatizing the behavior of all nondeterminate systems is.far. too complex to be-treated in this 
thesis. Only the types of nondeterminacy that. erise.in connection with the structure facility 
for the date flow machine will be treated, | 


The principal type of nondeterminacy. that will arise.in packet memory systems — 
is the removal of the requirement that the RTR packets.be returned in:the same order as the 
READ packets thet geve rise to them. For exemple, the input history. 


WRITE(1,11) ; WRITE(2,22) ; READ(1) ; READX2) could result in 


The system MEM is too simple to display this sort of nondeterminacy. For. example, MEM 
would return RTR(1,11) as soon as it received the first READ packet. It would not yet “know” 
that it was about to receive e second READ packet which would give it the option of 
producing its output packets in either of two orders. Later, we will exhibit implementations of 
systems which can meaningfully teke edventage of this nondeterminacy. For now, we will just 
have to accept that such implementations (that is, descriptive specifications) exist, and 
examine the form that the functional specification for such a system might take. 


2.1.1 FUNCTIONAL SPECIFICATIONS OF NONDETERMINATE SYSTEMS 


A nondeterminate system can give any af several legal output histories in 
response to @ given input history. The "function" defining the system's behavior is therefore 
multiple valued. One way to handle this situation is to treat the behavior of a system as being 
defined by a relation instead of @ function. The method to be used here, which is completely 
equivalent, is to use functions whose values are sets of output histories. For example, in the 


system fing, that we are developing, 


fran WRITELL, 11) 5 WRETEC2,22) ; READC1) 1 READI2)) = 
{ ( RYMEL,LED 5 PRER,22) >, CRTMED,2E) 5 RTE) ) } 


The situation may arise thet ((X} is empty fer seme X. This means that X is not a 
valid input hietery, and the-behevior of the sytem le undefined. This is different from the 
situation in which an illegal input gives rise to 2 well-defined “error® response (packet) from 
the system. An “error” packet is certainty more desirable than saying the system behavior is 
sent, ere: so pathatagicsl they must ‘simply be snstined notte occur. Furthermore, at some 
levels of dete in the description of « system, it is convenient te ignore error conditions if one 
can prove that they won't occur when the system ie functioning preperty. 


A furctinnat Gesaription oF: a ‘nenmeterminete syutew is: teretore 8: Cetirson of 

@ function which maps input hieteries inte sets ot output histories. It is usually most 

convenient to describe it ae « predicate defining whieh histories are in 1(X) for a given X, and 

that prodheste to-cfton t-tiieah AND ot « senor ob Other pretinetes, ot the functional 
deecription looks Sher 


¥ te in 100 it 
OGY? and: 
P,OCY} otc. 


The rule for reclization of @ function is ee fellewe: A. syetem resigns f if, given input history 
X with ((X) nonempty, it will eventually preduce seme ootput hielery ih 100. 


Tas eestieiy oleae fanetinee seciees ba cemenreate eysherme et obey a 
monotonielty property ae tellowe: 
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NONDETERMINATE MONOTONICITY (ND-MONOTONICITY) 


If Q and P are input histories end Q > P, then for 
any output, history X.in ((P), if f(Q) is nonempty there 
is a history Y in f(Q) with Y > X. 


Roughly speaking, this means that receipt of a legal input symbo! will never 
make the system unable to proceed Jegally, The purpose of the qualification “if £(Q) is 
nonempty” ‘is to allow for the possiblity thet en Wagel loput pachat. might. make the system. 
unable to proceed. 


We can now give the functional specification for the nondeterminste memory 


- fenwen 


If X = input history end Y = output history, 
Y te ir fuga? It 


(1) ¥ consists only of packets RTR(--;--), and 


(2) For all addr, the number of READIadde)’s in X = the 
number of RTRieddr,--)'s in ¥, and 

(3) For al addr end K, the K™ RTR(adde,-+) in Y, If it exiate, i¢-RTR(eddr,val) 
where lest WRITE(addr,--) in X before. K* READ addr) in X 


te WRITE(eddr,val) if such a WRITE(addr,--) exists, or val = 0 
if no WRITELaddr,--) exists. before. the K™.READIadde) in X 


The system NOMEM hes the property, that the-dete returned. in a. RTR packet is. 
the data in the memory (that is, the data in-the most recent. WRITE command. addressing that : 
cell) at the instent of the READ command corresponding to the.RTR...At-the instent the RTR 
packet is sent out, enother WRITE command might have already been-received, but that WRITE 
will have no effect on this RTR packet. 


Output: | RERAL) SERA.2) 
——> time 


Mt the instant the tirst HOR pavhat was tetunadl, @ WRITE command changing 
the deta from 1 40 2 tad aemadly ibemon given, tort Wh tani Yo, nequires thet the velue 
1 be returned. 


Here is © Tough wulline wt an imploneetution xf 2 aydion tat ewsliane Carne, 2 


SYSTEM «1 (roctining t04) 
(1) When 0 WRETE commend comns in, write-an te word of wanery intently. 
(2) than 2 READ aammend comes in, tech the wand irom sepanry imetertly, 
term 0 RIM sacange, ond gah it tet winilter or quews.— 7 
@) Tete nssages out ot dhe tutor avd cohen thom on autpat pactests at any 


time and in any ardac, subject to the eackrictions that: 

fe) every pantat in the toutter is eveckallly senowell, — 

(bo) whenever ¢:pechét te romped, 1 unt be the whiect in 
the inutter among thoue with tts wer aukdenes ‘hut is, 
the dnutior is tieet-in-tiret-out {F0O) saith nanginct to 


The jptementation given shove stil requires ‘that operations on the memory be 
inctentenseus, so tt is wat very usetal because Ht dosent tehe edveritage of the delay between 
a READ packet ond the WFR packet ‘ttidt results. “Yiu date in the WIR pocket must be the 
contents of the momory word at the instant the REMIYRER tttervat‘nastins. We would like the 
system to be site to wen the <n ot the mmeny ‘nial wt any ‘natant wing, the READ/RTR 
interval. Mere de on enenple of 2 2ystom thet takes auth fberty: 
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SYSTEM #2 (purported realization Of trmnary) 
(1) When » WRITE command comes in, write the word of memory instantly. 
(2) When a READ commend comes in, put. the message. READ(addr) in the 
Pending Read Butfer (PRB). 
(3) Take messages off the PRE at any time end subject to 
the seme restrictions es before, namely that every. 
message is eventually removed and the buffer. is FIFO.on 
eech address. When the message READ(addr).is taken from the 
Pending Read Buffer, fetch the date from memory and form 


(4) Take messages off the FRB at any: time end in any order 
_ subject to the some restrictions ot before, form a RTR 
packet, end send it es output of the system. 


This implementation does not realize fuour, -In the packet timing greph after 
the definition of fie,» the first RTR packet might have value 1 or 2.if this implementation is 
used. (The second RTR packet will always heve data velue 2.) 


We might like the system to take even more. liberty, by performing memory 
writes, es well as reads, whenever it wishes. Such an implementation might. be as follows: 


System #3 (purported realization of fyyey) 
(1) When a WRITE pecket comes in, put the message WRITE(addr date) 


on the Pending Write Buffer (PW8). 


(2) Same as (2) in System #2. 

(3) Take messages off the PWS subject to the same restrictions. 
as before, and write the data into memory. 

(4) Same as (3) in System #2, except that there is an additional 


restriction thet ne. message may be: taken from the PRE s 


Pe Ne 
(8). Some a6 (4 it Syaheme 62, 


Thie tus talle ts resties tse, . Newever, both Syetem 02 end System #3 do 
Fest2e tana Ht no WRITE packet le ever sent to the eystem when.eny READ/RTR treneactions 
are in: progress on that: word, That 1s; before s WHITE. pastet ie sont, s RTR packet must have 
been received for every READ packet sent addressing that word. ‘Fortunately, it is not 
difficult to. guarantee thet: thie requirement is: metr I le sisigty @: nondeterminate functional 
specification for the “rest of the world’, which we wil call: te “waer*. . 


Definition: The yser of «system is that to which: the 
the user are the eulpul ports of the given eyetem, and vice-versa. 


It would of esuree be totally ussiess 1: require thet, in order for 2 realization 
Of frases 10 Werk ite Uewr must resize © determinate functions! apecification. In fect, the 
user of @ system should have se few resirietions on ite behavior as possible. Such 
restrictions caw gerwrally: be specified by requiring that the user realize some nondeterminate 
function, just: as: ee systont Resell dees.: Tout ie, the difference between system specifications 
ered var spocitioctions te netting but # matter of degree ot resviciveness. 


The requirement thet NDMEM's user nol cend a WRITE command when any 
READ/RTR transactions are i progress can be met by requiring it fo realize the following 
nondeterminate functional specification fo csaere ! | 


 oneciauscn 


If Y = input history of USER and X = output history, 
(note the exchange of input end.cuiput so that X and Y 
refer to the same packet streams in both the syeten: and ite user) 


SE a SPA ng hngemiati bas oa 


then X is In fycassameslY? if 
(1) X consists only-of packets READ(--):. and WRITEG<,--) 


(2) For all addr, for any WRITEteddr,~-) in ¥;:the number of 
READIaddr)'s preceding it. in K:le ¢ ——: 
_ Of RTReddr,--)'s in.Y . 


The function frowewassen i easily seen to be ND-monotonic. This is because the 
restrictions on the user's outpul.X never become more-.steingent es:¥ Increases. As Y 
increases, the propasition "the number of AEADieddeys :preseding t:indt-is < the number of 
RTRaddr,-~)’s in Y° never. gose from true to fales; so-the:cet! of legal arrays X-does ‘not 
decrease. (If the "<° hed been rapleced-by "=", it would nat-he:Mimenctonic.)  . 


While system #3 does. not by iteplf realize fi 5 lt dows restizs fi, if 
connected to e user that realizes fipcy To prove this, the important step is to show that each 
READ(eddr) packet generates a RER. packet containing: dete defined, by the most recent 
WRITE(addr,--) pecket a aici atta aaacacel stream. 


Let te = the instent ohn the READ addr) pechat ¢ comes-in. There mey be © 


pending WRITELagdr,--) packets in, the PW at.t,. If there ere tiéne, the-most recent 
WRITEladdr,--) packet in the input stream hes slready passed out of the PWB and into the 
memory unit, so its data is In memory word eddr. If there ere WRITE(addr,--) packets in the 
PWS at t,, the most recently inserted packet there. is the most-necent .WRITEladdr,--) packet 
In the input stream. Therefore, letting 


deta th yang WATE in he PO tne ¢ 
D,,, (t) = if there is such a packet are ce 
| ti casta ‘Gh ard ge tna every oc nek 


° 


we must show thet the date to. be. eventusliy: returned ina: RTR: packet's Dsa;(to)- lett, = 


the instant when.the READXedde) packet Jeeves the ARE: First, we chow that D._. (t) does not 


change from t, to t,. Since. the. READaddir) packet bes iantored the'system, it fas left the user. 
Since the corresponding RTR(addr,--) packet has not yet been generated by the system (and 


tt] 


won't be unti after t,) it has not been received by the weer. Therefore, there is » READ/RTR 
transaction pending on addr, 00 the user ie not sending any WRIiTEfeddr,--) packets. 
Therefore, whichever WUTEledir--) peshet in te PWR is youngest wil stay youngest as 
long ac it stays in the PAR Gn-se-tong ac-there-are any WHET Xeddr,--) packets in the PWB, 
D4 dees not change. As tong ae.there sre ne WTBtedr,-~} packets i the PWB, D.,, = the 
contents of memery, which dosen't change either, becouse only removet of 2 WRITE(addr,—-) 
peckst from the PWH cen change the contents of memory werd addr. | 


There can be ne transitions from ne WRITE etd) packets in the PWE to one 
or more pachete, becaven-the wer ic not conding any. The-rasteiing case te consider Is the 
disappearance of the leat. WiUTEeddr,--> pechet from the PWR’ This packet is clearly the 
youngest, so D_,, (just priar t:dleeppeerance) = he date tn the packet. This data is written 
into memory by rule 3 of the implementation. O_. (just efter disappearance) = date written 
into memory = deta in the packel that dueppesred Therefore 0, (t.) = 0... (t,) 


At time t,, when the REAOteddr) packet teoves the PRB, there are no 
WRITE(addr,--) packets in the PME, by role @ oF the tnplhewhtetion Therefore D..(t,) = 
D,,,(t,) = contents of memory word eddr at t,. But when the READ\addr) packet is taken from 
the PRG, the memory werd te reed, end ite dete gave into # RTAédir;--) packet in the FRB. 
Thet pocket i: acetoes:HENMadA 1,2, sndte ie pallet Wl oventithy bo returned 
to the user. 


This example demonstrates « gereral principle: 


Whether or not a given implementation of » system realizes a 
Oe ere ee rane weenie tee ere crs ‘user 
realizes some other specific function. 


There is no way to get around this fect. There ere systems that correctly 
realize useful functions (even completely determinate functions} whet-connected to systems 
that obey certein rules, but behave in « totally patholegical wey otherwise. Furthermore, the 


sieeve! Se Pe ete Te RRREE SI 
elgg Re Rr 
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system often can’t tell whether the user has broken the rules. In the case of system #3 | 
above, the system would have been eble to tell whether: e WRITE{eddr,~-) pecket came in 
while a REAQ/RTR a ee ‘the me has 
no way of knowing whether ite-user:is misbehaving. 


The structure controller and packet memory eystem for a date flow computer is 
such a system. Perhaps the most important example of the structure controller and memory’s 
dependence.on the behavior of their user is the:-reterence count ‘ind garbage collection 
problem. The rules thet the. user Ge, the date flow computer) must obey In-order to assure 
correct reference accounting are as follows: 


(1) No pointer to a structure may be duplicated without giving a 
. Command to increase the rafesence count. 
(2) No commend. to decresse the referante count may be given 
unless « pointer.4e discarded. 


These rules guarantee that the reference count for a.node is at least-es great 
as the number of pointers to the node conteined enywhere in the computer. (Actually, the 
rules will be such thet the reference count le.gxegtly equal:to: the manber of poitters to the 
node. However, the penalty for too: high a reference counbis simply ‘thet « useless structure 
fails to pease ve mae space.) : 


Now suppose the einen (that is, the -structurecontroiier’s end memory’s 
user) violates the rule and.aliows the reference count to become: too. small. Everitually the 
reference count may become zero while a pointer to the-node stilt exists. somewhere. When 
the count goes to zero, the memory system recieims the nede-end:puts it:on the tet of free 


Two possibilities then arise. If an immediate attempt is made to use the 
“spurious” pointer to the call, in a SELECT instruction for example, the structure controller will 
send a READ command to the memory. The memory will know thet this is an illegal command, 
thet is, that the user hes violated its specification. It can then signal an appropriate error 


a2 


condition in ordar to prevent the computation trom gheing an incorrect result. 


It, on the other hand, the cell is remeved trom the free storage list end used by 
the structure controtier te auld seme new-elrushure ty the tew-the spurious pointer is used, 
there is no wey the memery can tell thet « vielation hes ecewred Wt hes ne choice but to 
POceet ne einen Seneeen Oe eee Sek cee tene mn Se re ere 
which ie compleiely diferent trem what wes intended. 


This ia nat 40 say thet the-dute flew computer hae ne way to check for errors 
in the handling of reference counts. Methods of doing an will be Gacussed in section 5.0.6. 


2.1.2 MUTUAL CONSISTENCY OF FUNCTIONAL REALIZATIONS 


Suppose a system realizes f,,, contingent on its user reslizing f,,..5 , which the 
user does H the original eystem:-cociines:- 6, « Gane it tellew thet the realizations actually 
ole th specs, th at aig tn ert Ltt pam be § nd. Each is 
the other's veer. 


M6 eny vieiation dees cour, there must be 2 first instant of violation. That is, 
there is an instant {, whan it fret becomes true thet-one system (esy $)‘hes'an Output history: 
which does not legally follow from its input history. There le s detey, however slight (even if 
it is only the deley caused by propagetion of electric currents through wires) in the behavior 
of S. Therefore S's output hielory at t, depends on T's output history slightly before t, , at 
time when T wea not malfunctioning, eo $ connot blame Its malfunction oh T. Even if S and T 
-both malfunction et precisely the seme inctent, nelther’S ner T knows about the malfunction of 
the other et thet imelent, and 20 neither melfunction can te:excused. I fellows that, if both 
systems conditionelly obey their functional specifications, they will obey their specifications in 
practice. 
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2.1.3 MONOTONICITY OF FUNCTIONAL SPECIFICATIONS. OF-THE USER 


We now give an exemple of how. not to:detine the functional specification of » 
user. Suppose the system MEM hes destructive readout, so thet it requires that the user 
rewrite eny data thet it reade... Suppees further that for some:reucer.the same deta must be 
rewritten, and that it must be done immediately, that ie, no other transections may take place 
et eny address between the.reed-end the rewrite, Here is encattempt at a functional 
specification for USER, Since USER dosen't know what:date to:write until it receives’ the RTR 
packet, we will require the rewrite to be « consequence of the RTR. | 


fuser 


Y = input to user, X = output from user 


For all addr end iif the." RTRledde) exiete.in ¥ andde: RTRteddr,date), 
then the i* READ(eddr) in X is immediately. fotiowad i X by WRITE(eddr data) 


Unfortunately, this does. not require. the user-te wait for the RTR packet after — 
sending any READ, not sending any more packets until the RFR ersives. ‘Far example, the user 
‘might send 


( READ(1) ; READ(2) ) 


Until the RTR(1,date) packet comes beck, the user has not broken eny rules. 
When the RTR(1,deta) does come back, the user will have retroactively broken the rules and 
be unable to do enything about it. Since we would like to simplify as much as possible the 
task of proving that systems obey functional specifications, we need to make the 
specifications reflect the types of decisions that systems make in practice. It doesn’t make 
sense for a system to perform some operation or emit some result packet on the basis of an 
input packet not having arrived end not being about to errive, 80 fp.» , as given above, is 
unreasonable. 


Ro) 


The pratiem is ‘that 4 iore 38 WO NDemoratonic. Lecicsa this, refer to the 
notation: eer ee see Poniie Mee ote 


Put QR ete)” a) 
X = (RBADGL) ; READE) . ne 


Wee 0262 gf i ry tg, rn 
>. 


NY =inpaldo:ueer, X«oulpuldromuser 


Fer all addr end i, the i” .READIaddr) in X, df 4 exists, is 

[ senmadiotety followed in X day WAIVE talite sate) 

it theme ie ont” S0PRtnddir,~) in ¥ anil tt is RTReddr date) 
| det dn 3X Hf theee-is net” Reddr -~)in-¥ 


This ie easily ween 40 0 NB-monstonic. 


Soman tapas came bps te ARPA BER: ED StH + 
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2.2 PACKET ACKNOWLEDGMENTS AND SAFETY 


All of the systems considered so fer. heve had to respond to incoming packets | 
however fast they were sent by their user, end there: was no limit.to the rateat which the 
user could send them. In the first implementation of MEM, the memory unit has to accept the 
commands directly, end hence has to operate at. unlimited. speed., System #3, implementing 
NOMEM, seems a slight improvement in that it only hes to put the commands into its buffers 
infinitely quickly, until one reelizes thet unless the memory. unit itesif ie infinitely fest the 
buffers have to be infinitely lerge. 


This is cleerly unecceptebla; no interconnection of speed-independent modules 

can make such assumptions. The problem ie one of setety. Ne packet may be sent until its 
destination is ready to receive it. The safety problem arises at several levels in date flow 
computers. Here we are concerned with it only at its most microscopic level. The solution to 
the problem is to acknowledge each packet trenemission. That is, for each port transmitting 
date, there is another port trenemitting acknowledge packets in the opposite direction. Every 
data packet must be acknowledged before the next data packet can be sent on the same port. 
We will require ali ports of all systems to have such en acknowledge port. 
(Even systems which would be safe without acknowledge ports will have them. 
This is because of the manner in which packets are transmitted. A packet transmission is 
indicated by a zero to one transition of ». “request” signal. An acknowledge signal from the 
receiver is needed to tell the transmitter to reset the request signal.) 


The implementation of the system MEM may be modified to acknowledge input 
commands only after the transaction on the actual memory unit is completed. This will make it 
impossible for the user to send a command while the memory is busy. Of course, the output 
port must also heve acknowledges, since the system to which the RTR packets are sent might 
be slow and need to be protected against overruns on its input. So the algorithm for AMEM 
(MEM with acknowledges) might be: 


(1) If a WRITE packet is received, update the memory (take your time!) 


and then send an acknowledge on the inpul echwewledge port. 
(2) If a READ pocket to received, tote dete from tne memery ent est 

& RFR packet out. 
(3). 1f i atheieinas io wadcadl os toeaaa walbonaes oak 

cond on somomndpe onthe input anlentediga pert. 


eisaig eeu: sscieaaaanaia 1 ard tea ante. 


Tramemtesion: of esinowledye puchele ie behavlordly stuiter to trenemizsion of 
_Rormal pachete, end con be hendind iw the seme way in: the-apaeilieation ofa system. Thet is, 
input ports, ant-vtep-veren, ‘Whe opstew AMEN hae tee inpiet porter the “tee” input port X 
ond the: culput ectwewtetge pert Y,, sate tp ‘er Veal" eat peit' ¥ and the input 
eee ery: 7 


‘ome 


input ports =X, ¥, cutmt porte ~ ¥, X, 


(1) FY] © number of READs in X 

(2) ¥, = RTWeder stated where the i READ in X 
te REACodar) anid the test WIT Eadidr,-~) vetore it, f there is 
one, be WTR aser fete tate there Wr ETB) 
before the i” RES: 

duly ee 


(0) P= 1S P< Ba 


It 16 easy te prove thet the: giver: implementation reuthiwe parts (1), (2), (3), and 
(4) Of frygyy - (It te very simiter to MEM) Parts (4), (), ent! (6) constitute the “Standard 
Acknowledge Restriction” that we will require all systems and aif users to obey. 
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Standard Acknowledge Restriction (S.AR) - weak form 


If X is an input port and X, is its acknowledge port, . 
(1) Xq congigts only of “ack” 
(2) yl < Pel 


It Y¥ is an output port and Y, is. its acknowledge port 
@ MSN. ae 7 


Given that a system and its user both obey the week form of the S.A.R.,.we can 
easily show that they obey the following: 


Standard Acknowledge Restriction (SAR). - strong, form 


If Z is an input or outpyt port ond 2, is ite seknewindge port, 
(1) Z, consists only of “ack” 
(2) Zl SS IZ) +! 


Proof: If Z is an input port of the system end an output port of the user, (1) and |Z,| < jZ} 
follow from the SAR on the system (letting Z =X) end Zi <4ZJ.¢.) follows from the SAR, 
on the user (letting Z = Y). If Z is en output port of the system end an input port of the user, 
just exchange “system” and "user". 


The SAR. is clearly NO-monotonic and hence sdmleaibie ae part of # functional 
specification. i a ke ve serge, foot 


In any proof that a system reolizes a function, it suffices to show that it obeys 
the weak form of the SAR contingent on its user obeying the strong form. 


We can now prove that AMEM realizes parts (5) and (6) of fey» that is, the 
S.AR. in strong form. 


Let Y « output of AMEM and input to user, X « input te AMEM snd output of user. 


First, number 6f WRITES in X 
= number of ache sent on X, in consequence of (1) of AMEN's implementation 
© i, - number of seks sent on X, in consequence of (3) of MAMENTE inplementation 
= Mal - Mel . 


Now [Y] = number of READS in X = Gay (2) of ANE a Iplementation) 
* X] - number of WRITES InN (by well-behevedness of wor) 
= Ki- K+ (derved shove) 
Sle Releits = @remGAR for user) 

SMSN Io 


Also ,| = number of WRITES 1X © {¥,) (derived shove) 
$ number of WHTEs in X + Iv] (from SAR. for weer) 
= rumber of WHITES in X' + number of HEAD: in % ty (2) of ANEM's implementation) 


This proves the week form of the SAA, from which the sireng form follows. 


2.2.1 CANONICAL PACHET COMMUNICATION 


Since the Stenderd Acknowledge Restriction narrowly limits the way 
acknowledge ports ere handled in the functions specification ef @ system, it le not uncommon 
for the Rendliing of fhe echnewledse ports te be viuttarly fndied in the Inplementation of the 
system. Wherever pessitie, cystom implementations will receive ond trunemit packets in the 


following way: 
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Canonical Packet Reception (RCVPKT) 


(1) Wait until a packet has arrived on the input port (it might have already arrived by the 
time this step is executed) take its data 
(2) Send an acknowledge for it 


Canonical Packet Transmission (XMTPKT)- 


(1) Send the packet 
(2) Wait for an acknowledge 


These operations will appear in. the system implementation language as 
“functions” that take port nemes as arguments and appear in assignment-statements. The data 
conveyed by the = is the contents of the packet. Assignment statements containing these 
Operations are like input/output operations in ordinary computer programs in that they “hang 
up” the program until the pecket communication has taken place. "Var := RCVPKT(port)” waits 
until an incoming packet hes arrived (and then acknowledges same). “XMTPKT(port) := 
expression” waits until the transmitted packet has been acknowledged. Programs may use 
multiprocessing as long as no RCVPKT or XMTPKT operations can be simuliteneousty executed 
by two processes on the seme port. 


It is easy to see that any implementation using the RCVPKT and XMTPKT 
Operations obeys the Stenderd Acknowledge Restriction. 


| Systems need not use these cenonical operations in order to be correct. For 
example, the implementation of AMEM given previously did not. That is why the proof that it 
Obeyed the Standard Acknowledge Restriction was so complicated. 


Here is an implementation of CMEM, a system whose behavior is similar (but not 
identical) to AMEM: 
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process starts at A 
input port X 
output port Y 
array memory init 0 


var command, addr, data 


A: command := RCVPKT(X); 

if command = READ(--) then 
let command = READaddr); 
data := memory(addr), 
XMTPKT(Y) := RTR(addr,data) 

else 
let command = WRITE(addr,data); 
memory(addr) := data; 


goto A 
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2.3 LATENCY 


CMEM and AMEM behave differently in a subtle way. Suppose the user 
transmits a READ packet and then refuses to acknowledge the. RFR .pecket that results. AMEM 
refuses to acknowledge the originel READ, and the entire system comes to # heait, since the 
user can't send enother command packet until the previous one was acknowledged. CMEM 
acknowledges the READ packet anyway (it heppeng:eutomaticellyes pert of the RCVPKT 
operation). It then refuses to ecknowiedge eny further commend. packets until: the RTR is 
acknowledged, because it gets hung up in the statement "XMTPKT(Y) := RTR(addr,data)”. 
CMEM behaves as though it hag an input buffer capable.of-storing one peckst. 


vs cellar cellabinirsdsiis Me funiiaen Sola Lines 2, 4, 5, end 6 of 
the specification of fuer, [eection 2.2) apply te OMEM elea: Linee.1 ond 2 are different: 


Syme 
(1) [Yl = number of READs in X 
(3) Ky = [gl + BX} - number of READs in X 
fowem 


number of READs inX if X]=Oorl . 
(1) [Yj = or ( KI 2 2 and [Y,| 2 number of READs in (x - last packet) 
number of READs in (X - lest packet) otherwise 


DX if PX) = 0 oF 1 
(3) KI = or (tl 2 2 and Ya) 2 number of READS in (X- lst packet) 
MX] - 1 otherwise 


This illustrates the fact that correct analysis of the latency of a sytem can be 
cuits complicated and requires careful analysis of the algorithm. 
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The ‘only Uiftevence between AMEM end GMEM arises if the user fails to 


acknowledge all-RTR peckets, that ts, if 11,| # VL If4¥,|-~ [one cen easily show that, for 


4) -~ nanan et HRABe tn K 
a i 


‘(To-prove this ‘fer: saauasecicetallanackal | Pe, wieiuaesdid 8s *< neeter ‘of READs 
in (X - tastspacket) cenvtameur:) 


The tatenay of -o-eyetum is ‘the nuit ot Cenmditds that it can accept and 
acknowledge whose vesdite fave not seen achnewletiged by the weer; that is, the number of 
pending commande ‘that it ven Wormer”. paarerineannenedieientuliass behavior, 
the concept of tatency i net-eeey to detieeprucietly. e 


Ove ‘system fer which it cen be deplined tc the FIFO, or first-in-first-out buffer. 
A FIFO of tergth ‘NC (end teving fetency ‘N) is a eyetem ‘with one teput port end one output 
port, which recliues tm ddentity fonction and acknowletiges ap to fiers tpuls then tts user 


hee acknowledged outputs. The function recived by 2 MFO wt tength W ic: 


ty 


(1) ML min CL Nghe 


)¥,~%, 
sla iabembiad iit 


A Ne ae ees eee ee 
following ‘program: 


procesess start st A, 8 
output port Y 


yar m 
var p init 0 | queve pepuletion 


A: until p # N doy 
k := RCVPKT(X) 
store k at end of queue; 
pp +l; 
goto A; 


B: —_until p # 0 do; 
m := item taken from front of queus; 
XMTPKT(Y) := m; 
pp- i; 
goto B 


For N = 1 this becomes: 


process stects at A 
input port X 
output port Y 


var P 


A: P := RCVPKT(X) 
XMTPKT(Y) := P; 
goto A 


A FIFO of latency zero cannot be implemented by any system using the RCVPKT 
and XMTPKT operations, though it cen be implemented witha few places of wire. 


Appendix I contains a proof that a series connection of FIFO’s of lengths M and 
N yields a FIFO of length MeN. 


When systems differ onty in their Istency, it is sometimes possible to make them 
equivalent by adding FIFO's to verious ports. Fer exemple, it can be shown that CMEM is 
identical to ANEM with a FIFO of tergih-one one input. if & couk! be shown that every 
system X is equivelent, except fer latency, to a syutem K, detined as heving latency zero, then 
the letency of the system X veukl be cheracterined by the fungthe-af the FIFG's that would 
have te be added to the various ports of X, te mates & identical to K. A system of latency zero 
would have te be one which never scknewledges any iqitt pectet unill uf resulting output 
pechets have been sent end acknowledged. AMEM 4s such & system, 20 CMEM could be ssid to 
have lstency 1 on its its input pert end zero an its euipul port. Ht te not clear whether such 
an ansiysis can be applied to nendoterminete systems of significant complexity. 


2.3.1 ARBITRATORS, DISTRIBUTORS, AND ALLOGRTORS 


Teree beck systems ere very portent in the design of the structure 
controler end memery, as well as other places in dete Bow vompeters. 


The arbitrate: i: 2 nendeterminate eystem with Mi inguls and one utput, which 
transmits seach incoming peuhet te the output. The order of the pastels trom each input must 
be preserved in the output ctream. The order i the eutput stream of patneld trom dtl 
ports ie arbitrary, in any resvoneble inplementalion it would depend on whith input pecket 

fret. An arbitrator restioes the teftowing tunttion, in which port number is indicated 
by 2 superscript inctoed of 2 eubscript: 


beeic (zere fetency) arbitrator f,., 


tt x', x8... x" ere inputs and Y is output, 
a xt... Hae 11 Fel WS. Ye 


1) Me min (SO Looe 


Vet 
- BN SITs cena 4 banely Nee ko Pat tig eetets t:% 
| £9) 4 CLAD W LO) = BA, the snepenee <>, OD... a> 
ts a eubsequence of Y. 
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Each incoming packet is tagged with its port number so that its source can be 


identified in the output. This identification feature is used in a few, but not all, applications of 
the arbitrator. 


Arbitrators are the major component ot the arbitration network of the data 
flow computer. The principal. use of the arbitrator in the. structure memory is to allow the 
address space to be divided into small pieces, with a separate memory module handling 
transactions on each piece. The LOAD packets, sant beck from the several modules are 
merged in an arbitrator, 90 that the entire interconnection of modules beheves as if it were 


One memory system. 


Arbitrators of nonzero latency may be defined as zero latency arbitrators with 
various FIFO buffers on the ports. Such arbitrators. ore, useful in various pleces throughout 
the data flow computer, but there is one place where the erbitrator: ‘must have latency zero. 
This is in the transmission of packets from the structure controiier to'the themory. When the 
structure controller receives an acknowledge for « packet it has sent ta the. memory, it must 
know that that packet is aheed of any other peckets that might subsequently be sent to other 
input ports of the arbitrator on that. memory unit. This probiem.will be-expleined in section 
5.0.4. 


An arbitrator of zero latency may be reelized by the following program: 


process starts at A 
input ports X,... Xn. 
output port ¥ 

yer p, input 


A: wait until a packet is available on any input port, 
let p := thet port; 
| this is nondeterminate! 
input z= the packet on port ps | do. nat acknowledge yat 
XMTPKT(Y) s= <p , input>; . 


send ecknowledge on port p; 
goto A | 


A distributer ts « determinate systom with one input and N outputs, which 
transmits incoming packets fo the output port selected by « deta field in the packet. Incoming 
packets are secumed'le be Uf the form <port, date. ‘The detributer strips off the “port™ field 
in the fine! result. . An Reectpnt Gitrovtor relies the towing functions 


if X is input and Y', v?,... ¥" are outputs, 
(V1, V8. Ey) fag OG Voy VE.» VED 


(1) Wi e@TLN, (V] number of pacnets 4,-> in X 
ad ae 

(2) Bil = 5 Mi 
jet 


(SH VIV), Yi= date where |* packet > in XK te 4, dete> 


process sterts af A 
inovt port x 
outeyt ports Y,... Vy 
A: wait until » packet te avaiable on port Xj 
2 = the packet on port % | do not acknowledge yet 
ligt z = <port , date>; 
XMTPKTCY,..,) = dates 
send acknowledge on port X,: - 
goto A 


Higher latency distributors may be defined in terms of basic distributors and 
FIFO buffers. a 


Distributors ere the principal meorent of the distribution network of the data 
flow computer. 


An allocator is a nondeterminate. variation of e distributer which transmits 
incoming packets to one of several output ports. Each packet is sent to. any output port that 
is ready to receive it, that is, any port that has ecknowledged all previous packets sent to it. 
An allocator is normally used to send packets to @ group | of identical units, always selecting 
any unit which is not busy. The structure controller of a date flow gomputer will typically be 

realized in the form of several identicel units in order: to incregse throughput. Operatidn 
packets from the instruction cells will be sent through allacetors to the. structure control units. 
(In fact, the other functional units of a date flow computer will be hanvdied t the seme way.) An 
N-output allocator realizes the following function: 


‘basic (minimal latency) allocator fatoc 


If X is input and Y!, v2..." are outputs, 
cv! v2, vx) € fara Yh vive. YA) it 


(1) > v= pl 


ist 
N 
(2) Kamin (KI,N- 1+ 5° it} 
| 
(3) Y', y?,... YY are disjoint subsequences of X 


It may be implemented by the following program: 


processes start at A, B 

_ input port X 
output ports Y'... vN 

_ queue q size N init (1, 2,...N) 
var pop init N 


A: wait until a packet is available on port X; 


z = the packet en port % | de act acknowledge yet 
k := item at heed of 

pap pop ~ 1; 

cond pechat x on port ¥*; { dew't wait for acknowledge 
until pap # © do; 

cond schnawtedge on port X,3 

gots A; 


8: walt until acknowledge i evelisie on any port, , 
jet p »= thet pert; 
| nendetorminete! 
tehe the acknowledge from port Y5 5 
put p tend of s 
pop » pep + i; 
goto B 


The besic sliecator given sbove doss not have latency zero in the sense of not 
acknowledging any input until the resultant extput bes been acknowledged - such an 
errengemect would detest the silecstor's purpose. The sysiom given sbove does heve the 
minimum lelency thet makes sense. 


3.0 THE BASIC MEMORY MODULE 


In this section # formal specification of the memory module "MM" will be given. 
MM Is the fundamental building bigck of the packet memory systam.. Each Mu system-is 
memory, somewhat like the system NOMEM described earlier, which handles. a specific ect. of 
addresses. To increase total information transfer rate, the address space of the entire packet 
memory system may. be divided into. smatier pieces, with ne. JM, unit. handling each piece. 
The MM units are connected through arbitrators .and. digtxibytors and form.a system which is 
itself an MM. This is shorizontal” composition, and is quite sjeiia. to the interleaving found in 
conventional memory systems, To increase, the speed on ingividyal traneactions. an MM unit 
may have a cache module "CM" connected to it. MM.with CM connected: to it is.iteelf en MM. - 
This is “vertical” composition, and ts quite similer to the cache memories found in high 
performance conventional computers. 


MM has one input. port CMI (“command in") taking commend packets from its 
user, and one output port RESO (“result out") returning reeults to the user. The memory — 
space is divided into words or calls (the. terme witl be. used. interchangeably), each of which 
corresponds to one node of a structure. Every memery transaction refers to one word, and 
every incoming or outgoing pecket bears the eddress of that word in its address field. The 
memory space is the same size as the address. space, and: the size is known to the user, so 
there can be no “nonexistent memory word" error. In most implementations, the memory size 
would be 2" where the addrags, fleld ot every pachat-le.N bite. 


Notation: FET‘*) means any of FET, FET”, or FET*. LOAD'*) similarly. 


Each word in the memory contains a data field and a reference count field, 
which are used by the structure controller ss described. in.section 1.2. LOAD'*) end UPD 
packets have corresponding fields. Furthermore, FET'*) packete-heve a tag field, which is 
returned unchanged in the corresponding LOAD) packet. 


3.0.1 LATENCY AND INITIAL MEMORY CONTENTS 


The specification of MM to be given belew dees not sey anything sbout latency. 
This is because Wid's user ic required to scknowledge évery result pecket. When this 
happens, MM will echnowledge every commend packet, régerdiess of whet its actual latency is. 
Hence, en sccurate description of Mis tstency is unnecsesary. | 


Initial memory contents witli else be left unspecified. In the functional 
specification of » memory, the definition of initiel contents aries it the specification of the 
system's response to READ command thet was net preceded by e WRITE. The specification 
of MM ett sesume thet this dees net eccur. In ar ectudl data How computer, a free storage 
ist will be generated when the system starts, which requires writing on every cell. 


3.0.2 INFORMAL BEHAVIOR OF Mu 


There ore 5 types of input peckets te MW, end 4 types of output pockets: 


LOAeddr, deta, ref, tex) . 
["ref” is the reference count) 
FET* (addr, tag) increases the reference ceunt by ene end returns 
) LOAD*(edde, deta, ref, teg) 


' ["ref* is the reference count after the increment] 


FET (addr, teg) decresses the reference count by one end returns 
LOAD (addr, deta, ref, tag) 
CLR( addr) (“clear”) waits until all FET/LOAD, FET*/LOAD*, and 


FET /LOAD™ transactions on the indicated word have 
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completed, and then returns DONE(eddr) 


UPD(eddr, data, ref) ("update") writes the date-and reference count 
into the addressed word. It returns no result, 
and hence uses no tag. 


MM is nondeterminate as was the exemple memory NDMEM, in that result 
packets referring to different cells are not constrained to. appear in the same order as the 
commands that gave rise to them. MM is further nondeterminate in that it may rearrange 
LOAD'*) packets referring to the seme cell. Such nondeterminacy would.not have made. sense. 
for NOMEM, since RTR packets with the same data end same address were indistinguishable, 
but, in the case of MM, LOAD'*’ packets may have different tags. 


Since LOAD) packets involve a change of reference count and may be 
reordered arbitrarily, the question arises: What happens te the reference counts appearing in 
such peckets if they ere reordered? The answer is that the result packets have reference 
counts consistent with their own order, not the order of the original command. packets. 
Example: Suppose the reference count of cell A is 1, and the command sequence 

FET*(A, T1) 5 FET*(A, T2) 5 FET (A, T3).3 FET (A, T4) 
is sent. Some of the possible results ere 
LOAD*(A, D, 2, T1); LOAD*(A, D, 3, T2) ; LOAD™(A, D, 2, T3); LOAD™(A, D, 1, T4) 
or 
LOAD™(A, D, 0, T3) ; LOAD(A, D, -1, T4) ; LOAD*(A, D, 0, T1) ; LOAD*(A, D, 1, T2) 


The reference count temporarily becomes negative! 


The reference count appearing in any LOAD* packet is one more than the count 
in the preceding LoaD‘*) packet. Similarly, the count in a LOAD” is one less than, and the 
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count in # LOAD is equat to, the. count in the preceding LOAD'*’, Some implementations of MM 
will never reorder LOAD? packets referring te tte sun adiiress, stthough they may reorder 
those for different atdrecess.. H this is te cate; tw reference count will never become 
negative, which removes the. neud for « sign bit hh the referenee count field. 


When the user gives « CLR command, it must not send any further commands of 
any type for the: indlested cett, untll the corresponding GONE pactel hes returned. (The 
purpose of the CLR command ie fe lew out pundits fransections. W would defeat its purpose 
to continue sending: commnunds:) : . 


Like: NOMEM,. Mil requires: that no UPD commend be given while any 
transactions are pending on the indicated cell. 


3.0.4 FORMAL, DEFINITION GF NM ANG NEUSE 


Thwew definitions do not show tatency or mate any reference to acknowledges. 
The user is required to scknowledge every result pechet and Mit is consequently required to 
acknowledge every commend Both systems of course obey tlie Standard Acknowledge 
Restriction, The definitions do net consider tive poueibility of itlegat packet types or invalid 
fields in packets. Ail univervel quantifiers ore intended to range: ver a set that is in each 
case obvious from context. . 


Note: in rules 2,3, and 4 the zeroth DONE in Y means the beginning of Y. The 
N+i* DONE in Y, where N © the: number of OONEs iv Y; muerte the ond of Y. Sinilarly for CLRs 
in X. The intention is te let the DONE snd CLR packets breek up X end ¥ into intervals, which 
makes it convenient te: think. of the entire Hetories as being precetiod end followed by DONE 
or CLR packets. 
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tes 
If X is input sed Teele Tear 


(1) Fora ar, th mmr of OO pce in. « he cum of CUM 
packets in X 


(2) For all addr, K, and tag, the number of LOAD(eddr,--,--,tag) packets between the Kk" 
and K+1" DONEXeddr) in Y = eee eee eee ore 
and K+1* CLR(addr) in X 


(3) For all addr, K, and tag, the number of LOAD (eddr,--,--tag) packets. between the 
K™ and K+1" DONE(eddr) in Y = the number of FET-(addr,tag) packets between the 
K™ end K+1™ CLAladdr) in X 


(4) For all addr, K, and tag, the number of LOAD*(eddr,--,--,tag) pachets between the 
K™ end K+ DONE(eddr) in Y = the number of FET*(eddr,teg) paenete between the 
K™ and K+1™ CUR(eddr) in X 


(5) For all addr, J, and K, the J'* LoAD'*)(addr,--,--,--) in Y is 
LOAD'*(addr,date,ref+D,--), where the last UPD(eddr,--;--) before the J'* 
FET‘*addr,--) in X is UPD(eddr,date,ref) and is preceded by I.FET'*\eddr,--) 
packets, and D = {number of LOAD*(eddr,--,--,--) packets} - {number of LOAD™ 
(addr,--,--,--) packets} among the 1+1" to J” LOAD \addr,--,--,--) packets in Y. 


‘maser 
111 Je laput to sumer ond X 4s outa, 4 faa aeagft) Ht 


(1) For oll addr, either the number of CiMaddr) packets in X= the number of 
‘D0MEiaddr) anckate-in ¥, 00 clan thave is he-utre ChMtaade) in X thin DONE(addr) 
in ¥, ond there are ao FET addr.) or PDair) packets after the last 
CLRhaddddr) in X. 


(2) For oil addr, for any URCadde,--,--) in -X, the number of FET*Xaddr,--) packets 
preceding it is < the number of LOAD!™ addr, --,---~) packets in Y. 


3.0.5 INPLEENEATION GF dhl 151NG A RANDOM OLORSS DEMICE 


Implementation of Nid with @ rendom access device is quite eaty. Assume the 


process sterts at A 
input port CMD 
var command, addr, dete, vet, tag 


A: command » ROVEKRCMDI) 
if command = FET(--,--) then {FET --return LOAD 
igt command. FET (addr, tag)s 
XMTPKTIRESO) = LOAD addr, mem-detaleddr), mem-refleddr), tag) 


else if command = FET (--,--) then | FET -- decrement ref end return LOAD™ 


mem-ref(addr) := mem-ref(addr) - 1; . 
XMTPKT(RESO) := LOAD" (addr, mem-data(addr), mem-ref(addr), tag) 


else if commend = FET*(--,--) then | FET* - increment ref and return LOAD* 
let command = FET'*)addr, tagh . 
mem-ref(addr) := mem-ref(addr) + 1; 
XMTPKT(RESO) := LOAD*(addr, mem-data(addr), mem-ref(addr), tag) 


else if command = UPD(--,--,--) then | UPD - update memory 
command = UPD(eddr, data, ref); 
mem-data(addr) := data; 


mem-ref(addr) := ref 


g 


ese | CLR - return DONE 
let command = CLR(addr); 
XMTPKT(RESO) := DONE(addr) 


goto A 


66 


3.1 HORIZONTAL INTERCONNECTIONS OF “Mid” SYSTEMG 
The functional speciticetions of ibd end its unser have the usefid properties that: 


(1) fay Od fase ore inveriont under reordering of commend packets 
referring to different words. That ic, such 9 reordering will not sffect the 
from the weer. ies 


(2) tras 97d Sipamey are simiterly iwartent under reerdering of result packets 
referring to differant words. 


(2) figs 97d Siegen ore invariont under reordering of L000! packets for the 
reference counts are suitably adjusted. es 


(4) the behavioral properties of MM and its weer are completely independent 
for different words. 


Property (4) makes it possible to connect MA systems and their users through 
distributors and erbitraters, and ctill heve an UM system. The following connections are 
possible: 


fuaassen the lerge dashed box resizes fyaugcy for larger address space If the user of the 
large deched bet: Yoalions frame » each small box'es veer realizes ‘yaanen 

‘For this to work the distributer and arbitrator must handle address fields 
property. If there are 2" smell Mil units, the addreds field of the interconnection is N bits 
longer than that of the units. The distributor picks out N bits of all incoming address fields 
end uses them ss the output port numbers. (For interleaving purposes, it might be most 
effective to pick out the least significant bits.) Those bits do not appeer in the address fields 
of the packets thet are sent to the MM units. The erbitrator inserts the input port number of 
each incoming packet into the address field in the seme positions as the bits that were 
removed by the distributor. 


This connection is one of the methods by which the transaction rate can be 
increased. Random access memory devices have the property that every read or write | 
transaction causes the device to become busy for some period of time, during which it cannot 
handle any other transactions. For example, a MOS RAM might be busy for 500 nanoseconds 
during every transaction, and therefore be able to handle 2 million transactions per second. 
Putting « FIFO buffer on it will increase its letency (as the term was defined previously), but 
its transaction rate stays the seme. The only wey to increase the data rate is to use many 
memory units. If a distributor cen handle 64 million packets per second on its input port, and 
an arbitrator can handie 64 million packets per second on its output port, it might be 
reasonable to use 32 MOS RAM’s, each in a separate MM unit. These are connected to a 32 


port distributer end « 32 pert arbitrator. The average rete at which packets come out of 
each port of the distributor is 2 silion ger second, attish ie the rate st which individual units 
can hendle them, Assuadng the comments sop antiornty distibated aver the address space, 
this interconnection will handle $4 sillien dransadiions gar escord. The retrieval delay for 
each item will til $e 580 nanoseconds, but that is an wmeveidable consequence of the 


For this interconnection to work eftectimely, the tatency of the individual MM 
units, or the output tatency of the distributer, smut tee et east ona, and preterably more. If 
the MM units and the distributor ati have tetency sere, the distributer wil! be unebie to 
echnowledge 2 commend, ond hence unite to got the anat ana, safl the commend hes been 
completely precessad by the thd sit. This would detest the surpace of ising mitiple units 
In proctice, the istency slight te somewhat ene then ane, in ender te maintain 2 trencaction 
rete near the wenimum in the presence of nenaiionn steligiied tnagmncy sf commends for 
cach unit. This can be aoxomplished toy plocing 2 email FIND butler tartween the distributor 
end sach WM unit. 


Multiple user connection 


This is just like the multiple memory connection, but with the roles of MM and the user 
exchanged, If the solid box realizes f,4, , each of the interfaces at the top of the diagram 
realizes toms for a smaller address space. If each of the users of this interconnection realizes 
tyeassen » then the collection of all of them slong with the arbitreter and.distributor realizes 
fyaassen 0” the large address space. 


As in the previous case, the arbitrator must map the input port number into a 
larger address field, and and distributor must remove the. corresponding pert of the address 
field and use it es the output port number. Each of the interfaces at the top of the diagram 
realizes on equivalent address space, end each uses 0 different subset of the memory space 
contained in the actual MM unit. 


This connection would be used if there were several users, each presenting 
commends at such ea slow rate that one memory module could handle all of them. Such a 
situation could arise if several cache modules are used which have a sufficiently high “hit” 
rate thet the rate of memory requests arising from cache misses is low. 
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3.2 VERTICAL COMPOSITION AND THE CACHE MOOULE 


In the section we describe the cache module “CM” which connects to an MM 
system end, so connected, realizes an Mid system with the seme address space. 


eee eS Po 


Cd me hewmen wean I 


$f the email box tebetied AAA rodlons i. the terge deshed box reslizes f,,, . 
It the user of the large deched box rectizes faasse » the wer Of the enti’ box reciires 
, . 
MeaER ° 


Vertical end hertzontel intercormections may be sized as in the following 
exemples, stnos in each case the wyelen being implemented s WAL | 
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_ The purpose of a cache is to retain the data of a small subset of the main 
memory’s address space, and return requests for data in that subset directly without reading 
it from main memory. Since the cache hes much less date then the main memory, it can be 
built out of fester circuits end devices without being prohibitively expensive. Hence any 
request for « detum that is in the cache (a “cache hit*) le antewered very quickly. If the cache 
te sufficiently well designed that it hes a high hit rate, the overall performance of the memory 
will be nearly es good as that of the cache itself. 


A cache must be designed to maximize the hit rate by holding those memory 
items that are likely to be addressed. This is usually done by assuming that the addresses 
being used vary slowly with time, end so, when an item is referred to once, it is likely to be 
referred to again soon, and should be placed In the cache. Therefore, when an item is 
eddressed which is not in the cache (a “cache miss”), the datum is fetched from main memory, 
placed in the cache, and also returned to the user. Subsequent requests for that datum will 
be cache hits. 


The size of the “items” that the cache contains sffect its performance. A cache 
for the main memory of @ conventional computer may use rather lerge items consisting of, for 
example, 8 consecutive words. This is effective because references to memory, especially 
instruction fetches, tend to be locelized in space. When a cache miss occurs on eny word, a 
block of 8 consecutive words Is read from main methory and loaded into the cache. Since 
references in the immediate future are likely to be In this block, the hit rate is increased. 
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The structure memory for a date flow computer hes no such locality: of 
reference. Therefore, the unit of cache organization will be The individuel word. 


Placing an item in the cache usually requires removing seme other item. The 


most populer strategy, end the one that will be used here, 4s the “lesst recently used” (LRU) - 


strategy. Each reference to # ceche item is noted in some sort of reference teble. When 
space must be made in the cache for 0 new datum, the item that hes been used least recently, 
that is, has gone the longest time without a reference, is chosen. 


When a write command is issued, the item in the ache is updated 
appropriately. in vome cache organizations, the Wom in main memory ie stways updated also. 
This technique, known es “write through”, will wot be weed here. Instead, the item in the 
cache will simply be marked as having been modified. When on item that hes been modified 
must be displeced from the cache, it is first written into main memory. This method has a 


lower voluwe of commande ging trom the ccho to main mewery than the “write through” 


method. 


It is crucial that the cache be sble to determine very quickly whether r or not it 
contains a given word. Since its memory epece is much smetior then the full address space, it 
must store the full address with each item. When a command is received, the cache. must be - 


searched for an item with the given eddress. It is importent thet the search be conducted 


quickly. . 


A popular method of Segaiieien the or for rapid searching is the “set 
associative” memory [12]. The cache is orgenized es an array of columns and rows. The full 
address space is similarly organized, with the same number of columns, and a presumably _ 
much greater number of rows. . Each item in the cache is constrained to correspond to the 
same column in the full address space as its own column in the cache, Therefore, to. se. 
for a given item whose full eddress is known, the address is seperated into row and column. 
If it is in the cache, it must be in the s. seme column as its column address in the real memory,’ 
80 only that column of the cache need te be searched. Furthormere, erly row addresses need 
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to be stored in the cache along with the items. The column addresses are implicit from the 
position in the cache. 


This organizetion works well for a suprisingly smelt number of rows in the 
cache. For exemple, the mein memory cache on the 18M 370/168 computer hes only four 
rows. (The number of rows je referred to as “ceche depth") Te determine whether a given 
item is in the cache, only four address comparisons need to be mede. These can easily be 


The column number of a word in the full eddress space is typically taken from 
the low bits of its. addrese. The row number comes from the iemsining bits. This allows 
consecutively addressed iteme to reside in the cache in-edjecent columne of one row. 


Example: Suppose the full address: space« contains 4096 addresses, and 
addresses consist of four octal digits. Een Se Oe ee en ee 
is the column number. The cache depth ie three. : 


column number 


row address 551 5560 543 504 464 425 425 425 
dete A B C D E€ F G& 4H 


row address 412 417 447 313 314 315 270 241 
data I J K tL M N O P.- 


row address 242 242 242 242 246 271 365 413 
data Q R S&S T U Vv Ww RX 


The cache holds the item with eddress 4472, with data "K". When a command is 
received requesting the contents of location 4472, the address is divided into the row (447) 
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and the. colunwr (2).. Golan: 2° of the: cache: is. thew: searched: for 447. It contains 543, 447, 
end 242. 447: ie.compared+wittt-these-thres: numiere-ciulteneeesty. TE metches the second of 
them, so. the date: ecsaciated! with it (I) -is:retummed?te: the: user. 


When: a: new: itent is: to: bee pat: isto: ther caatey. ite ooluens number is known in 
advaence,.s0- only: ite: row neusti be: determined: tyr esaschings tt ootenwr for’ the least’ recently 
used item. For exemple; if en entty: toe 2124:ntitieterunted, colts # ie: searched. If the 
least recently. used: itemise 314,1b is:removed:. [icite: Sneultj? Hit fe on) an: UPD packet is sent 
to main memory, . cortuining: the: eddress: (3144): and: the: dete (Wi; The row address is then 
changed to 212. 


The: determination: of: whiely: item: in: a: colmmn: wae: lowest! recetitty used can: be 
Whenever any: reference is. mats; thet tens: counter ie. set! te: zero: and sil others in its column 


Because each operation in the cache iwolves:enemination of an‘ entire column, — 
the cache memory: itself should: be: organized: so: that! eseh: colunw: ie: a “word", that is, the. 
entire: cotumr ie reed-or written: at: once;. 


3.2.1 DESIGN.OF Ci: 


The: funstional: specification: of Clitiis: very: shupler ee ae through 
its “top” ports. and: realize’ f aaseq. trrougit:ite “Botton” portts. 


fou 


If (CMDI, MEMI) = input ports, end (RESO, MEMO) = output ports, 


(RESO, MEMO) © fog {CMO1, MEMI) if 


(1) RESO € f,_{CMDI) 
(2) MEMO € fsa nen MEMI) 


An implementation of a system realizing foy will now be given. Each word of 
the full address space Is in. one of eight states denoted N, P, P’, Q, Q’, R, R’, and T. 


N - The word is not in the cache at all. (Since the ceche is much smailer than 
the full address space, most words are in this state at any instant.) There 
ere no pending cammands from the.user te. the system. There are no 
pending commands from the cache to the mein. memory. 


P - Space hes been reserved in the cache for the word, and at least one FET'*? 
has been sent to main memory, but no LOAD'*) has come back. One or more 
FET*).0AD'*) transactions are pending to the cache. Exactly the same 
transactions are pending to the main memory. 


P” - Same as P, but a CLR packet has been received from the user. One or 


more FET'*)/LQAD'*? transactions, plus e CLR, are pending to the ceche. 
The same transactions without the CLR ere pending to the main memory. 


Q - The first LOAD’) has come beck from main memory. A CLR packet will be 
sent as soon as main memory is eble to accept Ht. Zero or more 
FET‘*/LOAD™ transactions are pending to the cache. Exactly the same 
transactions are pending to the main memory. 
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Q - Same as Q, but @ CLR pocket hes been received from the user. Zero or 
more FET 2.900'* tworsactions, plus & CAR, are pending to the cache. 
Le eee eee eens Saree ee ore eee re, 


R - The werd le eo se FAS eect ey tb 
in progrese. in maint memory. A CL packet has bean sent te remove them. . 
No CLR packet hee been received from the user. Zero er more — 
FET‘*) /LOADt*) transactions are pending to the cocte.. The same 
meets: ae OG wn ronere ae ee eee 


R” - Same an R, bat CLF peciiot hus beew resetved fron: the user. Zero or 
more FET’? pone%* treneactions, plus & CAR, ave ponding to the cache. 
ee nee ee re eee eer 


T - The werd is truly he the cattle, Pe ee 
cache or from the casi terthe wee memnety, 


The normal stetes for ssiubcaiieiet dnamamtencatica fhe word is in 


word to underge transitions thet evertusily redull Iu Ne teing in afats T: If the commend is 
FET, the: word must be reed from maint menery, sipboletetontis 
intermediate states, IF the commend ie UPD, the wand ts created in tHe cache in in state 
cither cose, come other werd moy Reve tu be duplned, gsiny wow otte T te state If the 
“modify” flag. fev that wandie-on,. av UPS packet le sont te mate maimed). 


The specifications of NAF ond ite ser require thet the weer accept al! result 
packets from Wht hittle only remuired te sctept commends wien the resuifs of previous 
commands have luew. steupled by the user (sittieugh: at efficient inntemertation of MM might 


allow meny commends to be it progress at dren) Tie sin Wi eal sdee a | 


must sccept peckets from mein. memery, at MEM, ever: when mein memory refuses to accept 
eny further commends threugh MEMO. CM sometimes must wait for memory to accept a 
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command. While it is waiting, it mey refuse to eccept further commends at CMOI, but it must 
always be willing to accept packets at MEMI. CM may essume that eny packet sent through 
RESO will be accepted. | | 


The reason why CM allocates a cache cell for an item and puts it into state P ss 
soon as the first FET command comes from the user, is to avoid « deadiock, that is, « 
situation from which the system cannot proceed. If it simply sent the packet out through 
MEMO and did not allocate the cache cell until the first LOAD) packat.camp beck, it would 
use its own space more efficiently, but would be in danger of deadiock. (P cells are useless, 
since they do not contain data.) This will be explained in section 6,0. . 


Jn the following description of the cache algorithm, the manipulation of the 
counters to determine the least recently used item is not shown. 


STATE N 


FET'*\addr, tag) at CMDI - Create space in the appropriate cache column. 
Either use en empty space (this situation can only erie, when the, system is 
first started) or remove the least recently used item in state T. If no item 
is in state T, wait ‘until one enters stale 7, not. accepting. eny: packets on 
CMOI while waiting. (Items in other states will progress to state T.) When 
the item to be. removed is found, write it out if its “modify” flag is on, by 
sending én UPD packet. at ‘MEMO. If main memory is not accepting packets 
at MEMO, wait until it does. Then creete a new item in the cache with the. 
given address, "modify" = 0, state = P. Leave the date and reference count’ 
tlelds unspecified. Also, send a FET) pecket, identical to the incoming one, 
out through MEMO, to fetch the data. | 


CLR(eddr) at CMDI - send DONE(addr) at RESO. 


UPD(eddr, dats, ref) at CMDI - Create space in the cache as for Fer'®), perhaps 
sending an UPD packet to memory. Then create @ new item in the cache 
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with the given: eddrese, “modify” = 1, dete and reference count from the 
command, and state » T. | —— 


LOAD'*? or DONE at MEME - can't occur because no transactions are pending in 
mein. memery. 


STATE P 
FET *Yedkir, tagh at CMDI.- Send the seme packet at MEMO. 
CL R{addr) at CMDI - Change te state 


UPDtuder, date, ref} at CHEE - ~ een't happee, singe traneactione are pending in 
the cache. 


LOAD'*Neddr, dete, ref, tag) et MEMI - Dapesit the date snd: reference count 
inte the cache word, end sand the same packet eut.at RESO. If the main 
memery le accepting commends, send 8 CiMiadér) at MIND and chenge this 
cache How torstate ®. W rat, chenge to ste Q. . 


DONE at MEMI.- can’t happen, since no CLA hes been given te main memory. 
STATE P” 


FeT'*), UPD, ‘or GLA ot CHO - can't hepsi, ainse user hes « CLA/OONE 
Nenecton pee: 


LOAD'*taddr, date, ref, tag) at MEMI - Deposit the deta and reference count 
into the cache word, and send the seme packet out at RESO. If the main 
memory is accepting commands, sond e CLMinidt} at MEMO and change this 
cache item to stete RF’. Hiretcearenste sie 
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DONE at MEMI - can’t happen, since no CLR has been given to main memory. 
STATE Q 
Note: CM does not accept any command at CMDI whenever any item is in state 
Q. Qis simply a temporary state that is waiting to send a CLR(addr) out through 


MEMO and go into state R. 


FET‘*), UPD, or CLR at CMDI - can’t happen, since cache is not accepting 
commands. 


LOAD’? at MEMI - same as stete R 
DONE at MEMI - can’t happen, since CLR hes not been sent to main memory. 


Mein memory becomes able to accept a command - Send CLR(addr) through 
MEMO, change to state R. 


STATE Q” 
Note: CM does not accept any command at CMDI whenever any item is in state 
Q’. Q’ is simply a temporary state that is waiting to send a CLR(addr) out 


through MEMO and go into state R’. 


FET (2), UPD, or CLR at CMDI - can’t happen, since cache is not accepting 
commands. 


LOAD'*) at MEMI - same as state R 
DONE at MEMI - can’t happen, since CLR has not been sent to main memory. 


Main memory becomes able to accept a command - Send CLR(addr) through 


MEMO, chenge to stute Rr 
STATE R 


FET*Xeddr, tog) at CMDI - Update the reference count in the cache, and set 
the “modify* bit it the pechet wae FET” or FET*. Send LOAD'*\ addr, data, 
newret, tag) through RESO, where date end newret ere current contents of 
the cache. Note: at the instent this happens, there may still be 
FET‘# OAD‘? transactions pending in main memory. If so, these Fer‘*) 
packets were eerfier then thie one, but the corresponding LOAD'®) packets 
won't be returned unti later. This le the circumstance which causes the 
general system MM to occasionally return LOAD*? packets in en order 
different from thet of the FET® packets, 


UPD(eaddr, data, ref) at CADE - Update the cache, set the “modify” bit. Note: if 
on UPD pechet ie received white in state A, we know from the rules for 
NMIJBER thet no FET™7LOAD!! transactions ore pending in mein memory. 


CLR(addr) at CMDI ~ Change to stete FP’... 


LOAD'*Xeddr, dete, ref, teg) ot MEME - ~ Ignore the “ret” field in the packet. 
Increment or decrement the reference coun in the cache if the packet is 
LOAO™ or LOAD®. Do not set the “modify” flag, since mein memory siready 
knows sbout the reference count chenge. Send LOA MNeddr, dete, newrel, 
tag) through RESO, where newre! = the updated reference count in the 
cache. 


DONE(addr) at MEMI - Chenge to state T. 
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STATE R° 


FeT‘®), UPD, or CLR at CMD! - can't happen, since user has # CLR/DONE 
transaction pending. 


LOAD‘*? at MEMI - seme as state R 
DONE(addr) at MEMI - send DONE(addr) through RESO, change to state T. 
STATE T 


FET‘*addr, tag) at CMDI - Update the reference count in the cache, and set 
the "modify" bit if the packet wes FET” or FET*. Send LOAD'* addr, data, 
newref, tag) through RESO, where data and newref ere current contents of 
cache. 


UPD(addr, data, ref) at CMDI - Update the cache, set the “modify” bit. 
CLR(addr) at CMDI - Send DONE(addr) through RESO. 


LOAD‘*) or DONE at MEMI - can’t happen, since there are no pending 
transactions in main memory. 


3.2.2 PROOF OF CORRECTNESS OF CM 


A proof of CM’s correctness is generally similar to that of the system MEM 
given in section 2.0.3. The memory state required in the specification is the contents of the. 
last UPD packet in the input history. One must show that, for a cell in states Q, Q’, R, R’, or T, 
the data in the cache itself is the same as that in the last UPD packet et CMOI, and, if the 
modify bit is off, this deta is in msin memory also. For states N, P, and P*, the correct data is 
in main memory, that is, the last UPD at CMDI has the same data as the last UPD at MEMO. 
These properties must be shown to be preserved for all state transitions, end it must be 
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shown that all legal FET'*) commands will get the correct data. Furthermore, the effect of 
reference count modifications resulting from FET* and FET” commands must be taken into 


account. 
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4.0 IMPLEMENTATION OF MM USING A “ROTATING” DEVICE 


“Rotating” memories such as charge coupled device (CCD) or "magnetic bubble" 
shift registers, or magnetic disks, ere rightly considered to be essentially unusable for the 


main memory of a computer because of their excessive retrieval.deley. In e.date flow. 


computer, totel transaction rete is es important a criterion es ratrievel: delay, and so the 
disadvantages of these devices largely disappears, making.them perhaps economical as a mass 
store. On the other hand, further improvements in RAM. technology. may render these shift 
registers obsolete for most applications. This. section. is. predicated.on the sseumption that 
CCD’s or bubble memories will be economical and useful.in the packet. memory system. 


In a rotating memory, the data is structured.in a ring which “rotates” pest a 
“read/write head”. Equivalently, one may think of it as 2 fixed ring and a pointer rotating 
around the ring, with memory transactions permitted only on the cell currently pointed to. If 
the addresses of words correspond to fixed places on the ring, it is possible to predict when 
any given cell will be pointed to. Commands from the user cen be. stored in @ memory 
somewhat like a queue, sorted by position, so that the pending transaction at the head of the 
queue is always (or nearly elways) the one that the pointer will reach next. This will make 
Optimal use of the availability of date from the CCD. . 


There ere a number of CCD architectures currently in use. In the “line 
addressed random access memory” (LARAM), only 3 small part of the device shifts at full 
speed at any one time. The rest shifts and recirculates et a much lower speed in order to 
conserve power. The intent is to make the device beheve somewhat like e random access 
memory. To retrieve any One item, one finds the section in which that item is stored, and 
directs the CCO to shift that section at high speed until the desired:item is found. While this 
is happening, the other sections are shifting much more slowly,'so this architecture is not 
. efficient when many items ore being sought et one time. It is therefore not suitable for the 
type of packet memory system being considered here. | 


Two other types of CCO’s are the "serpentine", which is simply a long shift 
register (it “enakes” beck and forth on the IC chip), and the “serial-perallel-serial", which is 
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simply a collection of interleaved shift registers. These two types differ only in engineering 
specifications such es dete rate ond power consemption, They béth Behave like long shift 
registers, end hence ere suitebie for the type of memery under discussion. 


There are « number of implementation considerations thet must be taken into 
eccount in designing © rotating packet memory. Por exemple, ¢ number of shift registers, one 
for each bit of s date word, may be weed, oc thal s how date Word comes into position on 
each clock putes. On the other hend, 2 single shift register might be used, with each word 
stored serieity, or eny arrangement betwoer these two extremés can be used. One might also 
Use en unueusl correspondence between address and siiit régistéy position. AN of these 
considerations are ivretevant to the structure being considered, se we wil assume the memory 
is @ ring of full words, ordeved by address, with eddress sero following the highest address, 
and the pointer sconning the ring in order OF increasing stérest. ‘ny other implementation is 
equivelent te this. 


Pe eee rete ere ie ee er regerdiess of 
what type of device # actually is. 


Pending traneactione (thet is, packets reteived et CMU) are stored in the 
transaction list (TL), which is presumably much. smulior than the mimnery itself. The TL is 
presumably resized with « random access memory device. In order te evoid moving data in 
the TL unnecessarily, it has # ring structure: just Whe the memory. Trenéections are placed in 
the TL at or neer the seme enguter position se the pavilion it memory of the word to which 
they refer. ee ene ee es 
to many conescutive acdiréesee of momery. : 


Lot €X» be tte function mapping addresses in the entire eddress space into 
the corresponding address.in the TL. Thie is colled-the hueht faction fer reseons that will be 
expleined later. €X2 is just the integer pert of the quétiont of X divided by the ratio of 
memory size to TL size. In a realization in whieh aif cleus are powers of two, €X> is just the 
appropriate number of high order bits of X. . 


r 
When @ command is received for address X, the command packet is placed in 
the TL at address €X>, or the first free address thereafter it €X> is full. Assuming a 
uniform distribution of addresses appearing in commends, the Tt should be uniformly: filled. 


As the memory pointer’ rotates through the memory, snother pointer, maintaining about the 
same enguier position, rotates iivough the TL, picking out the next transaction to perform, 


The TL is organized much like the “ordered hash table" devised by Amble and 
Knuth [2] , with modifi¢ations to allow for its circulerity ‘and for the fect that items are being 
removed from it. In an ordered hash table, each item has.a hash address. It ‘is placed in. the 
table at its hash address or in the contiguous block of items after the hash address. This 
block is in increasing order of data value. This ordering mekes it possible to determine 
whether sn item is in the teble much more quickly than in a conventional: hesh table. 


Although ordered hesh tables are intended tor entirely different applications 
than the transaction list of a packet memory, the concept is well suited to this application. 
The “value” of an item in the table Is the word address: appeering in the packet.. Let a(P). 
‘denote this address for packet P, and call it the: "CCD address". The “hash address” 
corresponding to CCD address X is just <X>, defined earlier. (Hash functions are usually 
‘designed to be random, but thet property is not desirable here.) The hesh address of packet 
P is therefore <a(P)>. 


Because the'TL is a ring instead of a linear list, a different definition of order is 
needed. The concepts of “greater then” and “less than” are replaced by “clockwise from”. and 
“counterclockwise from*. Since eny.item is both clockwise and counterclockwise from eny 
other item, the order of two items must be defined relative toa third. This is done through 
the use of intervals denoted in ordinary. mathematical notation. -{X, Y] is the interval from X 
clockwise to Y. If X < Y,.it hes its. customary meaning.” It-X:> Y, {X; Y] is the ext of numbers 
from X up to the highest address, and then from zero up to Y. “Open” and “helt open” 
interveis have their customary meaning, that is, [X, Y) means [X, Y] exclusive of Y, etc. [X, Y) 
ont Lie are. Seti complemerse cheer per 2X ay, : 


The Ordering of hash addresses and word addresses is expressed in terms of 


whether or not an element is in an interval. Z € [X, Y) mans that if one sterts at X and 
moves clockwise, one reethes Z before Y. 


The general rule tor matnteining order in TL is thet, if one. gees clockwise from 
an item's hach address to the item test, one wil net sans ary engty.celle:and. will pass only 
“emator” tome, ttn, Heme whee beoh eddreseee are courterteciwive from tht one, This 
be one digit. The hah function pichs out the firet digit. The trasseution dst-has: 3 cette end is 
drawn os 2 circle. 


Colle O and 6 ere empty. Coll 2 contains « packet with address 16, whose hash 
address is 1 but was dlepleced because coll 1 is full. 


It le. possible for the transaction Net toconteln several pechets referring to the 
seme CCD eddress. Spseitinatny the faltowing contigurattons are ponoibte: ; 


One or more FET *) packets, When the COD: putnter reaches the eppropriate 
Sotho a a ae of 
Loan!) packets. 


One or more FET‘*) packets, followed by @ CLR When the COD pointer reaches 
the appropriate address, the LOAD) packets will be sent out, followed by 
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A single UPD packet. The date will be written into the.CGO: when the 
. appropriate address is reached. | 


No other states ere possible. This is bacause.it ie a vielation of figanee to send 
en UPD packet when there ere FET!) or CLR pastels pandiog. lf an UPD ie. given athen an 
UPD is eiready peadin the new, one. singly. replaces, the old.one,..Jt.0,FET{*).Je given: when 
oF peg te en en ern pe ot cones 
LOAD'*) packet. 


Intuitively, the rule for a well formed transaction list is that the fines 
progressing clockwise from a ceil to those items with thet cell’s hash address must never 
cross each other or pass over an empty ceil. If en item with CCD address 43 were placed 
into cell 6, this rule would be violated, since the line from 4 to 43 would cross the line from 5_ 
to 85. The insertion sigorithm must instead put the 43 into cell 5 end move the 55 to ceil 6. 
Furthermore, all items with the seme hash address must be ordered by CCD address. In the 
exemple, 16 is clockwiee from 11. 


To insert an item, stert at its hash address end search clockwise until an empty 
cell or a cell containing en item with higher (more clockwise) CCD address is found. In the 
former case, insert the new item. In the latter case, insert the new item after meking space 
for it by pushing the old item, end all those contiguously following it, one space clockwise. In 
the exemple, insertion of item 10 would require pushing 11, 16, 25, 32, and 55 clockwise. 
Insertion of 42 would require pushing only the 55. 


While incoming command packets are being placed in the TL by the above 
procedure, packets sre being removed snd sent to the CCD memory. This is sccomplished 
through the use of a transaction list pointer (TLP) which rotates clockwise roughly in 
synchronization with the CCD address pointer. When the the CCD pointer points to CCD cell 
10, the TLP points to TL eddress 1. Since a packet for address 11 is found there, it waits until 
the CCD pointer = 11, removes the packet from the TL, and performs the indicated operation 


on the contents of CCD address 11. The TLP is then immediately advanced to the next 
position, 2. Since the packet there specifies address 16, it waits until the CCD pointer = 16 
sineuaapeaimies wie cnmceuar odes ed araet Cogakcusiel The TLP the moves to 3 
and the prosees cantinuss. 


The removal of tems from TL mekes it necessary to modify the rules for a 
well-tormed transaction Net. Hf 16 is removed fram the ‘example list, the fine from cell 2 to 
item 25 passes through on empty coll, whith would vielete the sendition given previously. 
Theretore, the region trem which peckete are removed ts ‘dbtlore be be the “removal region", 
ond it is permissibte for the line fram on Hem’s Kesh adilhes th the lem itself to pess through 
the removal region. The removal region is delimited at its counterclockwise end by the 
“removel pointer” RP, and at is clockwise end by TLP. sik hier cancanee the example 
leohe the thie: 


removal region 
RP, 


as, ‘4 


Whenever an item is removed, RP. is sat to. the hash. eddress of that item. In 
the example, after 25 is removed, RP will be eet to 2 (25's hesh: address), and TLP will be 
advanced to 4, - 


The rules for a well-formed transection list cen now be given formally: 


(1) W j,k € TL address epace, if j # k and TL(j) # empty » TL(k), 
[ KATLEYD 5) €  [ CoTLIQ)D KJ. 
(That is, the interval from the hesh eddress of an item to the item itself is never 
contained within the corresponding interval for another item, i. e. the lines never cross.) 


(2)VjeC(RP,TLP) TLY) = empty 
(That is, cells in the removel region ere. considered to be empty.) 


(3) V j,k € TL address space, if TL(j) = empty # TL(k) and j ¢[ RP, TLP), 
J €[ <aTLK)>, &) . 
(That is, the interval from the hash address of an item to the item itself does 
not contain any empty cells not in the removal region.) 


(4) V j,k € TL address space, if Ka(TL(j))> = Ka(TL(k))> and j e[ <a(TL(k)>,k }, 
then a(TL(k)) 2 a TL()) 


(That ts, if two items have the seme hesh eddress, the more clockwise one has the higher 
CCD eddress, i.e. elf the peckets having one heshatkirese ere ordered by CCD address.) 


(5) V j,k € TL address space, if j e[{ CaXTLE)D,k) and af TL(j)) = a(TL(k)), 
then Vmelj,k} Tie) = af FL(j)). 
(That Is, all items with one-CCD address ere adjecent. This is necessary to be sure that, 
whan # sequence of adjacent FET‘) packets and e CLI are found, it is possible to 
return the LOAD'*) peckets followed by » COME, with né-denger that there ere unseen 
packets elsewhere referring to the seme CCB address.) 


(6) V j,k @ TL address space, if j oe w) and pla @ a TL{k)), 
then TUj)-weepleced in the teble before TLay: peo 
(That is; the iteméwith the seme CCD address are ordered by age, the youngest being 
most clockwise.) This property makes it possible te return e DONE packet as soon as 
a CLR is encountered in the removal scan, since the packets are encountered in the 


The insertion atgerithm requires seme cere when-pessing through: the removal 
region. If the scan sterts outside of the region end then-enture:the-region, the Hem:is placed 
in the first cell, and the region is: shortened by“one-ep- that thet‘celis ‘no'ionger part of the 
region. If the ecen begins inthe region but notvinats sret-eell,-the-eeen skips over the region 
and starts after its end. If the scan begins in the first cell of the region, it skips to the end if 
its CCD address is greater than or equal to that of the item just past the end. Otherwise, it is 
inserted in the first celt andthe region is: shortened.- 
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nin eri 
To insert ‘Do this: 

22-27 putt 3 sot RP im 4 

90-33 pute Sect RP 
30-95 «sé th at G, push the 26 and 43, 
36-42 = --—séiputh at Z, pauah the 43 
43-77, 00-07 put at 0 


The algorithm for inserting an item into the TL is given in appendix III A. If the 
TL slveedy conteins en UPD packet for the seme eddress, it instead performs the indicated 
action, perhaps modifying the UPD packet and perhans.transmitting.s packet et RESO. 


The removel algorithm is somewhat simpler. The. TL item: pointed to by TLP is 
next to be removed. The CCO pointer indicates the current item availsble et the CCO output. 
From the standpoint of the algorithms for bendiieg the Tl, the CCD pointer mut be considered 
to be inexorably advencing under control of an external agency. The.external. agency is the 
clock controlling the shifting of the CCO shitt register, or, in the: case of @- magnetic disk 
memory, it is the information being read from the disk’s timing tracks, . 


The fact that the CCD pointer is synchronized to external events means that it 


cannot be integrated fully into-e system using the packet communiestion principle. It must be 
considered external te the pechat system, and some eytithrenisers or arbitration devices must 
be used in the interface. The design of euch:an interiate is @ common problem of digital 
system design, end is beyond the scope of this thesic.. We: will assume that the interface 
between the synchronces memory device and the: peoketeystenqemalets of ports CCDI and 
CCOO. Every time the CCD advances to a new address, an ADDR pecket containing that cell’s 
address and date are sent to the system threugh port COOL. If the system fails to 
acknowledge the ADDR pashats fest enough, #0 thet the CCD is prevented from sending one, it 
may either drop the packet or weit until the COD hes shifted sit fhe way sround to the same 
address again. After the system receives an ADDR packet et CCD! announcing that an address 
has been reached, it may transmit « WRITE packet ot 0000, giving the address ond new date 
to write. If this packet is not transmitted soon. enmugh, t might be toe late to write the date 
into the CCD. in this case, the OCD shifts oil the way around, net omitting any ADDR packets, 
until the address Is reached again, and then witles the data. 


Wasting an entire rotetion time whenever the ssynchrenous part of the system 

cen’t keep up with the CCD clock may seem drastic, but it degen’t heppen very often. 
Whenever an ssynchronets system must conmunicéte with something such ss the CCD clock, 
there is the possibility that it mey he tate: However, it le not difficult to design the system 
such that the probability of this happening 16 vanisHingly small If this is done, it is possible 
to prescribe drastic remedies when it dees occur, without significantly degrading system 


The ebove description of the interface to the COD may be somewhat simple- 
minded. Many memory devices require that the write commend, end the deta to be written, be 
given before the previous date from the seme addruss is avellatile: This means that the 
protecol whereby the system issues @ WRITE packet only after receiving an ADOR packet 
bearing the dete might net be appropriste. ‘In thi’ céae of » COD or Other shift register, the 
problem con be solved by Having two “taps” oh'the regitter: Gis fer reading, end another, 
one or two bite tater, fer writing. In the cede a'disk memory, thé’ prdbiiom is more serious, 
and may require thet the diek announce: eaeh dddrevs slightly before the data becomes 
availeble. The necessary modifications to the asynchronous pert of the system will not be 


ett BE oe ae teeth, BMP ES 


treated here. 


The rotating memory module then looks like this: 


The removal algorithm waits for an ADDR packet at CCOI matching the address 
contained in the packet in the transaction list pointed to by TLP. When found, it performs the 
indicated transaction, perhaps sending a packet out et RESO. It then sets RP to the hash | 
address of the item which was just processed, which may shorten the removal region. The 
item is then erased from the trensaction list, and TLP is advanced to the next position. If TLP 
now points to an item having the seme CCD address, that item is processed also, using the 
same date. All transactions giving the seme address ere handled in this way. Any reference 
count changes ere noted, and the modified reference count is written back into memory with a 
WRITE pecket at CCDO. 


When TLP reaches a cell which does not contain @ transaction for the same 
address, either it is for a different eddress or it is empty. In the former case, the system 
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waits for the CCD to reach the new address. In the latter case, it sets RP = TLP, destroying 
the removal region, and then advances both RP and TLP, in step with the ADDR packets that 
give the CCD address, until it finds a transaction to perform. 


The algorithm for the rotating memory is given in appendix III B 


5.0 STRUCTURE CONTROLLER DESIGN CONSIDERATIONS 


In this section we will examine e tow of the considerations that must go into 
the design of an efficient structure controller. 


5.0.1 CHECKING THAT THE CONTROLLER OBEYS Figascen 


The structure controller never issues an UPD command unless: the. reference 
count is known to be one. Since this is so, there can.be no transactions. pending on that ceil, 
co the requirements of figases ere met. This ls contingent, of course, on the rest of the 
computer correctly realizing fowmoucmsen «A reference, count. vielation. by the computer 
could lead to an UPD packet being sent while there are transactions.pending. 


5.0.2 PRECISE REFERENCE ACCOUNTING WITH IMPRECISE REFERENCE COUNTS 


In checking that f,q, satisfies the needs of the structure controller, there is a 
point of possible danger that needs to be checked. Since LOAD!) packets may be returned 
from the memory in en order different from thet of the. FET'*) packats, it- was: shown in 
section 3.0.2 that the reference counts returned-from the memory may be unusual, perhaps 
even negative. Is it possible for. this to interfere. with the cell menagement mechanism? The 
answer is no, as long as the following rule is obeyed: 


After increasing a reference count (with # FET*), do not pass the result to any 
destination until the corresponding LOAD* has returned. 


For example, if an instruction cell indicates two destinations for its result, the 
reference count of the result must be increased. with a FET* before the.result is sent to the 
destination cells. If one of those cells is « SELECT that issues # FET~ ta-reduce the reference 
count, the FET* must ect first. Furthermore, it is not enough to.rely.on the zero latency 
arbitrator to be sure the FET* gets to the memory.before the FET”. The FET” must not be 
sent until the LOAD* arising from the FET* has returned. This is accomplished by not sending 
the result to the destination cells until the LOAD® hes been received. 


It is easy to see that ne coll wilf 'falt to be reclaimed thet should be reclaimed. 
At the time the fast “owner” ef a coll issues a FET” to discard it, there are no other 
Operations pending on the cel, co the LOAD™ pacts? thet le returned will heve the correct 
reference count, which is zero. | we . 


To see thet no cell will be sccidentelly’ rectaimed that shouldn't be, consider a 
cell with reference count 2, owned by instruction cefis X end Y. Suppose X performs a 
structure operation thet discards its copy, 60 thef'a FET” It issued. We must show that if Y 
does not discerd its copy, the LOAD” thet arises from X's operation wil not have reference 
count zero. The only wey the reference count could péssibly g0:to zero is if Y also causes a 
FET". Since Y dose net intend to discard its copy ofthe call; e FET* must have been issued 
first. (Thet ts, the reference count should actually go up te 3, then dewn to 2 and then 1.) 


The memory receives the following sequence ot CMOI: 
FET (addr, X) 5 FET "Cedar, Y) 4 FET"(edde, Y) 
Th station ob eho shat wich the sete and hed LOAD pacha are reverse 
LOAD" (addr, 1,0) 5 LOAD (adder, 0, Y) 5 LOAD" (eddr,--, 1,¥) 


This can't happen, because the FET"(addr, Y) ie not sent until the LOAD*(addr,--,--,Y) has been 
returned. es : 7 oes 


5.0.3 MEMORY LATENCY 


MM's latency wes left unspecified only for the purpose of proving correctness 
of MM and its user. When actually imptomenting a practice! packet memory, it may be 
necessery to build e high degree of tatency into some modules in order to obtain good 
performance. For exemple, a-“roteting” itplomentatiin dt KAA°using @ charge coupled shift 
register may be designed te have hundreds or thousands of commends pending at one time, 
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although its correctness does not depend on this. 
5.0.4 THROUGHPUT AND DISTRIBUTED PROCESSING 


"One of the fundamental principles of date flow computers is that, if enough 
parallelism exists in the program, a computer be able to run arbitrarily fast for a given logic 
speed. To do this, it must distribute the computation and be free of bottlenecks. If « dete 
flow computer could only have one multiply unit, that would be a bottleneck, since it would 
limit the rate at which multiplies could be performed. The. data flow concept must not plece 
any restrictions at all on the number of multipliers that @ computer can have (although eny 
given computer of course hes a fixed number). There must not.even be bottlenecks in ports 
through which packets must pass. If every multiply operation pecket. had to pass through one 
input port of an allocator on its way to the multipliers, that would be unacceptable, since the 
logic speed places a limit on the rate at which packets can pass through a port. For example, 
if a port could handle packets 100 times faster then # multiplier could process them end ail 
packets had to pass through one port, it would mean that no more than 100 multipliers could 
be usefully employed. . 


In the case of simple functional units such es multipliers, it is not difficult to 
avoid bottlenecks. Muitiple functional units may be used, and the arbitration and distribution 
networks that connect them to the instruction cells may be designed to be free of bottlenecks 
end thus maintein any desired throughput rate (5). For the same reason, multiple structure 
controllers sre used, each with its own ports connected to the erbitration and distribution 
networks of the data flow computer, Also, multiple memory units are used, because the total 
memory transaction rate is greater than can pass through a single: pair of CMDI/RESO ports. 


It is not possible to compartmentalize the structure operation facilities as cen 
be done with simple functional units. Connecting each structure controller to one memory 
module is not correct, because each structure controller must heve access to the entire 
memory address space. The structure controllers must be connected to the memories through 
an interconnection network consisting of arbitrators and distributors for packets going in each 
direction. Command packets from the structure controllers have pert of the address field 


a aie 


tie tet 


removed and used to select the output port of the distributer, just as was done for the 
multiple memory connection in section 2.1. In this way, each structure controller “sees” the 
full address spece, while each memory module supports only » email part of the total address 
space. The commend packets from the different structure controliers are merged in 
erbitrators, which append the incoming port number to the tag finid, se that the result packet 
will be returned te the correct controtier. Pachets coming out of the RESO ports of the 
memory modules pass through distributors thet use the added bits of the tag fietd, and 
arbitrators that use the incoming port number to reconstruct the full addrees. 


A2 inserts 


input port 
inte address 


D2 removes and 
uses part of 
tag to select 
output port. 


The treatment of address fields and tag fields is symmetrical. One could think 
of all pending structure operations as occupying a “tag spece”. dust as each memory module 
supports a small part of the total sddress spece, each structure controller supports a small 
part of the total tag space. The job of the interconnection network is to make the entire 
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address epace available to each structure controller, end te make the entire tag spece 
available to each memory: unit.. Broth) “S 


It is not necessary for the network to:piace the distributors before the 
arbitrators. Such a network would have 2 size proportional te the: product: of the number of 
structure controllers and the number of memory units, which may be excessive. It is possible 
to mix arbitrators and distributors in » network in such @ way that the: eize is reasonable but 
bottlenecks ere. avoided. | 


. Because UPD packets do not heve a tag field and do. not give rise to result 
packets at RESO, it ie. nacessery that the erbitrstore and distributors carrying packets from 
the structure controllers tothe memory modules(these labelled Al and D1 in the preceding 
diagram) have latency zero... This is -00-that, when ao structure controller receives an 
acknowledge for an UPD packet, it will be guaranteed thet the packet has pessed through the 
arbitrator end. is therefore: chead: af: eny packet thet may subsequently be: introduced into 
write on 2 cell, thereby completing the creation of a structure. When it receives an 
acknowledge for that UPD command, it essumes that the structure is complete, and so it 
returns it to the rest of the. computer. An instruction cell:in the computer; ‘heaving. received 
this structure, may fire, causing a SELECT operation to be generated. The allocator may send 
the SELECT operation packet to another structure controller, which then sends out a FET 
packet with the same address. If there is buffering before the arbitrator that merges packets 
from the two structure controtiers, the original UPD packet might still be in such a buffer, so 
the FET packet passes through the arbitrator first. If this happens, the old data will be read, 
rather then the new date supplied by the UPD packet. By making sure that the distributor 
and arbitrator have latency zero, the UPD packet cannot get stuck in a buffer. When the first 
structure controller receives an acknowledge for the UPD packet, that packet is known to 
have been accepted by the arbitrator, end hence it will precede any subsequent FET packet. 


If it is not feasible for the interconnection network to use distributors and 
arbitrators that have no memory, it is necessary to put tag fields in all UPD specification 
passing through the network. An “adapter unit” is placed between the netwark and each 
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“memory module. The adapter pesses aii packets through except UPD packets. When it 
receives UPOleddr, data, col, tag), 1 sends PDieddr, date, nut) to the memory and UACK(tag) 
beck to the interconnection network. The structuse cantrolier dass net return a structure to 
the rest of the computer until it hes received ACK sapiies ter all UPD commends that it has 
sent. Re ee ee ee ent eee ere routing 
networks end is heynad the ecope of this thesia. 


To maintain just one free storage fist would create a bottleneck, so each 
structure contretier hes ene. henever a etructure eeniesiier aseds a werd in order to 
create @ node, it thes ite etidracs trom the packet qeatauind at gat port UDI. (UID stands 
for unique identifier) The structure contraiier dena nat:ank for adiremes at (ADI; they are 
supplied in an “weeding” sivaom, 2 tect on thoy ateecimoutiged. 


i Arlee aia aes Baa maaan denice inca: of 
UIDO paris are coswesied te the 42DI ports thrangh « callection af aldedters end arbitrators 
called the.189 quiwerk, She gucpene of this network is to mnattinie-e cupply of free cells to 


UNI 
(from UI00) (to UID) 
UNI UNO 
(from UIDO) (to UIDI) 


Each structure controller, in. addition to performing structure operations, 
maintains a free storage list. Whenever. an acknowledge e recetved on UIDO, it takes a cell 
from the list end transmits it,in « UID packet: through UIO0. Since-a reference count echeme is 
used for recovering unused cells, the controller watches for. words whese reference counts go 
to zero. Every time it reduces a reference count by iseving e FET commend, ihexemines the 
LOAD” packet thet is returned. If it shows # reference count of zero, the word is reclaimed. 
This involves placing the word in the free storage list and, since whatever pointers it 
contained ere destroyed, reducing their reference counts if their elem bits-ere off. If either 
or both of the latter reference counts go to zero, these words ere reciaimed by the same 
process. 


The procedure is recursive, and is en unpleasent type of recursion because the 
completion of each operation cen produce two more operations to perform. Although the 
recursion always terminates, a huge smount of storage may be required to hold the list of 
words that need to have their reference counts reduced. The.problam at its worst can be 
observed in the case of a large tree, no subtree of which is shared with anything else, whose 
root node is discarded. All nodes have en initia! reference count of 1, so, when each node hes 
its count reduced, it goes to zero, making it necessary to reduce the counts of both of that 
node's offspring. 


To implement this procedure by simply issuing two FET” packets whenever a 
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word's reference count goes to zero (that is, whenever a LOAD js received bearing a count 
of zero), would creete an intractable deadiock problem because of the protiferation of packets. 
Instead, the procedure thet should be used is thet only the right offspring of « word should 
be trested at the time the word ts placed on the free storage list. The pointer to the left 
offspring will remein in the word while it is on the free storage fist. The recursion in this 
procedure is under control, since only one new operation is created for every operation that 
is completed. When « word is teken from the free shorage iist, the reference count of its left 
offspring is reduced, which may cause one or more wards to be reclaimed, before the word is 
used. 


The memory management sigoritim is as follows: 


(1) Whenever 9 word's: reterence count is reduced; examine the LOAD” packet 
thet is returned: it shows-e count of zero, pat the word on the free 
storage list and, 1 the elem bit in ite right thalt le zero, reduce the reference 
count of the wend peinted'te by thet fell, This ‘way cease this step to be 


(2) Whenever an: ecknowledge is received from pért LADO, get a word from the 
free ctorege fist and send the packet UNMiiddr, We tuft aif’ through UIDO. 
(The contents of the ‘eft half ere: sent eimply to aveld en extra memory 
reference.) 


(3) Whenever @ fresh coll is needed for creation of a structure node, teke the 
pocket UO(eddr, obj): at port IDI end acknowledge séme. Addr is the 
eddrose of the now cell. It the-olem bit of ebf te eff, reduce the reference 
count of the addressed word. This may cause step (1) to be invoked. 
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6.0.6 MAINTAINING INTEGRITY OF THE REFERENCE ACCOUNTING MECHANISM 


The possibility of an error. in the reference accounting end cell management 
mechanism is @ troublesome. problem, because, as. explained in section 2;1:1; it is. impossible | 
for the memory to detect a reference accounting error by its user. Furthermore, the effects 
of such an error are unpredictable, and may show up.in-completely unrelated parts of the 
computation, However, there ere.a few acd ea iad the probability of 
such an error being undetected. 


- First, all cells on the free storage list can be marked.in'some way, perhaps by @ 
bit reserved for this purpose. Any reference to a merked: sell: other than for the purpose of © 
removing it from the free storage list is a. detectable error. Also; the:free storage list can be 
orgenized in such a way that calis ere added. st one. end and remeaved.frem the:other, thereby 
maximizing the time thet a cell steys on the list once it is put there.-if a cell is erroneously 
reclaimed while a “spurious” pointer to it. exists, it will then probably still. be-on the free 
storage list when the spurious pointer is used, so the.errer can be detected. 


Another way of checking integrity. of reference counte is to conduct. an “audit” 
of the entire computer. This can be done at the end of the computation, and st any point 
during the computation. The host computer must disable all instruction cells and wait for all 
pending operations to clear out of the structure controllers. and.the-routing networks. All 
reference counts can then be checked egainst the contents of the input registers of the 
instruction cells. 


6.0 THE DEADLOCK PROBLEM 


The structure controiler and cache module that were described previously were 
both required to have s lerge capacity for stete information which would be unnecessary if 
one could slways be sure that the device lower in the Nerarchy would eccept a command. 


In the case of the structure controfler, the general behavior upon receiving a 
result packet from the memory is to perform some transtormation’on the data in its state 
memory and then send a new command packet. Its internal state meniory could be dispensed 
with, and the state information placed directly into the tag fields of the packets. When a 
result packet is received from the memory, = “memoryless” controller's functions would then 
be simply to perform @ transformation on the pachet Iteelf, forming #tiew packet which is sent 
to the memory. The reason thc fells is that one cont be sure the memory won't decide to 

return several result packets (perhaps elf pending ches) before it accepts any more c ommand 

packets. Suppese this happened to e memoryless structure controller. It would have no 
place to put the result pockets if the memory unit ien't’ accepting any more commands, so a 
deadiock would occur. The problem is that the ‘controtier hes vielwted the rule that it must 
always be prepered to accept the results of all pending operations. A structure controller 
Pe te errr eer nee ee eres ree of all 
pending operations. 


A similar problem arises in the cache module. If a word is not in the cache and 
a FET‘ packet is received, « coll ie immediately elloceted for it end placed in state P. A 
FET‘*) packet is also sent to main memory to fetch the data. Until the data returns from the 
memory, the cell in the cache does not have date in it, so it serves no useful purpose. It 
might seem to meke more sense to allocate the cache cell only when the first LOAD’ packet 
is received from the memory rather than when the first FET) packet is received from the 
user - that is, to bypass stete P altogether. The problem is thet the creation of a cell in the 
cache may require writing out the celi’s former contents. If the cell is created in consequence 
of the LOAD'*) packet coming from memory, the cache may have to send a packet to memory - 


. in response to a packet from memory. If the memory sends such Loap‘*) packets but does 


not accept any replies, the cache would have no place to put the dats, so a deadiock would 
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occur. The cache implementation given in section 3.2 avoids this problem by reserving space 
for the LOAD‘*) packet in advance. If an UPD packet must-be sent-to the memory, it is done 
in response to input from the user rather then from the memory. This way, if the memory 
temporarily refuses to eccept the UPD, the ceche can simply refuse to accept input from its 
user. : 


In both the structure controller and the cache, the cost incurred es a result of 
this problem is an amount of memory equal to ell the packets that can be simultaneously 
pending in all tower levels. In the controller, this is the: state information 'for afl concurrently 
executing structure operations. In the cache,.a call might be in-state P for every ; 
FET'*)/LOAD'® cycle thet is pending ot thet.inetent, Since a cell'i state P is useless, the 
cache must be that much larger than it otherwise would be, for a given level of performance. 


In the case of the structure controller, the memory space is needed somewhere 
in any case. If a great number of memory treneactions can -be: pending simuitaneously, a 
“rotating” memory, such as wes described in section 40, is predumebly being used. If a 
memoryless structure controller is used,.the state information for pending operations is stored 
In the tag fields instead of the controtier. But the tags.of pending memery Operations must be 
stored in the transaction list of the rotating memory, so whatever space was saved in the 
controller is used up in the transaction list. rs 


Why, then, would a memoryless structure controller be more desirable? The 
reason is that memory space inside the controller is much more expensive than in the 
transaction list. The controller must be sble to process intormation:es fest as the highest 
level of the memory hierarchy. If thet highest level is a cache using high speed (and 
expensive) devices, the controller must be equally fest. The rotating memory is at the bottom 
of the hierarchy, so its transaction list can use slower end less expensive logic family. 


. In order to use 8 memoryless structure controler or @ cache which does not 
use “P" cells, the memory system below the controller or the cache must obey the following 
“fixed latency law": : 
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Whenever a result pecket ic trenemitied at RESO, the device must accept a 
packet st CMI. if thet packet 4 an UMD, % must eecept yet encther, until it 
has taken one thet is rot UFO. Mato a even H the weer done not accept . 
anything further at REO. 


The reason UPD pechets ere a special case is thet they do net generate any result, so the 
system should be able to abeorb them in uniimited nenbers. 


Some memory systems obey this lew. A random excess Implementation of MM 
clearly dos. A rotating. implementation con ated, ewe tw tremmection Wet hes fixed size. 
Whenever en item is token out of the TL, enothur can be inserted. (The tmplenentation of the 
rotating memery given in section AD eid mat deve behave the wey, but M could ennity be 
modified to do 20.) . 


The systems thet de net obey fhe fixed letenty tew ere the horizontal 
composition of MM unite end-the-cache, The former dwhales te inturcehriection network 
between the structure controllers and the memory units. Inthe tase of the horizontal 
interconnection of unite cath of which ebeys the faed tetenty tow, ‘when one unit transmits 2 
result packet, it. will eccept a new commend: That redett pathet posses Through the arbitrator 
and becomes 9 result of ihe interconmection, 08 Ww itercennection must accept another 
commend. If the command is addressed te « different Mat ant than tft one that transmitted 
the recut, that unit might not be able to accept it. What to seeded te « way for the units to 
share the burden ot pending ireneustions with wich ether 


In the cose of the coche, maintaining 9 conetent rumber of pending transactions 
in the cache and memory. combined required waliteiiiy @ constant number of pending 
another command must ge from the ceche to mein mewery. However, such commands only 
occur wher there ere cathe misses. If the cache runs inte unusually good tuck and gets a 
continuous string of cache hits, it would: not-send comments te memtry. In brder to maintain 
constant latency, it would have to refuse any renadt pevtete tom mewory. This could result 
in some transactions remaining pending indefinitely. White this prebebly won't Céuse a data 
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7.0 SUGGESTIONS FOR FURTHER RESEARCH 


One of the principal problems remeiniig in the sree of the design of systems 
using the pecket communication principle in thw Gevelépinent Of & practical and systematic 
procedure for constructing modules that cen He prover 16 Méet given functions! specifications. 
An importent toot for thie teuk’ ie the dovelipment of « rigdrous anf concise Architecture 
Description Language (ADL). With the help of the ADL, the task car be divided into two parts: 


(1) Development of a proof methodology so thet systeme expressed in the ADL 
can be proven to meet functional specifications. 


(2) Development of a system construction methodotogy so that systems 
expressed in the ADL cen be constructed with confidence thet the physical 
device will reetize the ADL expression. 


For this purpose, the ADL must be simple enough to correspond neatly to the 
herdwere devices inveived, but powerful enough to meke proofs involving history srrays 
tractable. 


Another remaining problem is, of course, to develop functional specifications for 
all parts of the data flow computer system, including the structure controller, and give proofs 
of their correctness. The functional specification of the computer itself (that is, the structure 
controlier’s user) is needed, among other things, to show thet ne reference count violations 
will occur. 


An efficient structure controller needs te be designed, with special attention to 
the needs of programe that are likely to arise. 


The deadiock problem needs to be examined carefully, to see if it is worthwhile 
to build e memoryless structure controller. 
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APPENDIX I 
Proof that the concatenation of two: FIFO buffers is 2 FIFO buffer, end lengths are additive. 


This proof is given not because the statement is of fundamental interest, but es an example of 
the method of proving theorems ebout the bekevior of systems, showing acknowledgments in 
detail. 


Let a FIFO of size M have input-.port X and output port Z. 
Let another FIFO of size N have input port Z and output port Y, 
and let the ports Z and the acknowledge ports Z, be.jinked. 


xX YM, 


From the definition of the-first: FIFO, 


(1) [2] = min { DCL, [Zyl #1} 


(2) 7, =X, 
(3) Kgl = mein (2g) + 


From the definition of the second FIFO, 


(4) t= min (12), V1 +1) 
@)Y,=7, 
(6) [Zqh = min (ZI, Mal +N} 


Case I: Suppose |X| < IY,| +N 
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By the strong form of the. Standerd Acknowledge Restriction, . 


If (Z] = 1Z,) + 1, then, 
J} “Neh 
71s mI 
“ [Zl < Bl 


either § jiZj- es or it 


- eam, ence tip 04 ae 
(from 1) 


<- [al +N < BX], which is @ contrediction,.op we mush Ayave-{Z]:= 42.) 


“a= 

& Ma min {Ml +} 
Keyl = min { PCL, BX + M} 
Wil = Bl 
KIS) +M oN 


ee | = min { KI, IV.) + M+} 


Case Il: Suppose |X] > [Y,| + N 
Hf (Z| = [2,1 then 
Z| = XI 
<i, 


. aairaeaag naan ciah 
fet ‘Grom &: oe oa. CHATS gael 


(from 3) 
(gjuce M 2 0) 
(by hypothesis and. fact thet M 2 0) 


{from 1, since [2] # [Z,] + 1) 
(from 6) 


“> BS I¥,] ¢ 8, which is « contradiction, so wa munthave:(z)i=-2,| + 1 


PAR ARL. 

oo IZj=W.l+N+1 

= M41 $12 

= MeN +1 

IZ| < Kl 

sgl +1 < PO 

cho min (0, Ml +1} 
Xl = min (X, Nyl+ M+ N} 


In either case, 


pane taee ass 
(since N > 0) 


(from4).- 
(from 1) 


(from Sond t,)-= Wt + N) 


Yj = min { XI}, V1 +1} 
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Yi = x, (from 2 and 5) 
IX,l = min { XI, IY] +M+N } 


which are the conditions for the interconnection being a FIFO of length M + N. 
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APPENDIX II 
Algorithm for the cache. 


Actual lookup in the cache is not shewn. Instead, the special functions 
cache-data(addr), cache-refladdr), ceache-stelefeddr); end: ceche-~medieddr} ere used. These 
are treated as though they were arrays, and ere assumed to be defined whenever the given 
address exists in the cache. In-cacheleddr) returns true if the given address exists in the 
cache. 


Can-creeteleddr), where addr does not exist in the cache, tells whether it can 
be created, that is, whether some cell in its column is unused or is in state T. 


If can-create(eddr) is true, creation-cell-is-empty(eddr) tells whether the 
former case holds, and, if 90, cache-create(eddr) performs the insertion into an unused cell. 


Otherwise, cell-to-displace(addr) returns the address of @ cell in state T, selecting the least 
recently used item. Ceche-rename(oid, new) performs the replacement. 


‘ processes start at QA 

input ports CMDL, MEM 

output ports RESO, MEMO 

var cmd, item, addr, data, ref, old-addr, p 

ver m init faise | tetis whether to wait for input from MEMI 

ver memofiag init true | true when leet pecket sent at MEMO hes been acknowledged 
var memowait init false | true when need to send something on MEMO 


var wait-pkt | the thing to send 
var create-fiag init false | true when need to create 2 new cache cell 
var create-pkt | commend that led to creation 


var new-addr | address field of create-pkt 
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Q: 

wait for acknowledge on port MEMO, 

take the acknowledge; . 

memoflag := true; 

goto Q 

A: 

until memofieg or packet ts available on port MEMI. do; 

m := false; | becomes true if should teke packet at-MEMI . 


if memoflag then | is memory reedy for command?. 


if some-cell-ie-in-state-Q-or-Q' then —| see if need to send e CLR 
addr := address-of-a-cell-in-state-Q-or-Q; 
memofiag := false; 
send CLR(eddr) on port MEMO; py 
if cache-state(eddr).=."Q" then. | chenge.Q:to R, Q' to R” 
cache-state(addr) := “R° 
else 
cache-state(addr) := "R’” 


else if memowsit then —| see if need to send FET'* after creating a ceil 
memowait := false; 
memoflag := false; 
send wait-pkt on port MEMO 


else if create-fleg then | see if trying to create a cell 
if can-create(new-addr) then | is some cell in its cokann empty or in state T? 
create-fiag = false | yes, will create the cell 
if creation-cell-is-empty(new-addr) then 
cache-create(new-eddr) =| old ceil empty, just put in new address 
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old-eddr 1= cel-to-diepleceinew-addr) —_| find cell to displace 
if cache-mod(old-addr) then 

memofiag := false | write out previous contents # necessary 

send UPDioid-eddr, cache-deteteid-eddr), cache-ref(old-addr? on port MEMO; 
cache-renemelold-eddr, new-eddr) =| create the new coll 


| the new cache cell now exists 


if create-pht = UPIX--,--,--) then | whet command caused the creation? 
lat create-pht = UPD(--, date, reth adie new at eeerey 
cache-mod(new-sdde) r= truss © 
cache-date(new-addr) := date; 
cache-ref(new-eddr) := ref; 
cache-stetetnew-edde). °F" ee 
else | commend wae FET’) 
cache-mod(new-addr) := false; 
cache-state(new-eddr) := "P*; 
wait-pht :@ erest®-phhy | im sonmard rman to menery 
memowailt := true 
else , 
m := true | can’t create new cache cell, must wait - 
a | 
wait for packet en MEM! or OMDI, let P «thet pert. 
if p = ‘CMOP then 


| +++++ process packet from CMDI +++++ 


cmd := RCVPKT(CMDI); 
it cord © FET*Y=.,-=) then 
let cmd = FET'*Xadde; tag) 
if In-cache(eddr) then 
if cocha-stetaleddr) = "P* then 
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memoftiag := false; | state P, just send # onward 
send cmd on port MEMO 
else | stete is R or T 
if cmd = FET*(--,--) then | need to update reference count? 
cache-ref(addr) := cache-refaddir) + 1;- 
cache-mod(eddr) := true; 
XMTPKT(RESO) := LOAD* (addr, ceche-data(addr cache-ref(addr), tag) 
else if cmd = FET(--,--) then 
cache-ref(addr) := cache-ref(addr) - 1; 
ceche-mod{eddr) = trues 
XMTPKT(RESO) r= LOAD (addr, cache-data(addr), cache-ref(addr), tag) 
else 
XMTPKT(RESO) s° LOAD(addr, cacher-detaleddr), cache-ref(addr), tag) 
else | state N 
new-addr := eddr; —_| set flags so ceil will be created 
create-pkt := cmd; 
create-flag := true 
else if cmd = UPD(--,--,--) then 
let cmd = UPD(addr, data, ref); 
if in-cache(addr) then | must be state R or T 
cache-date(addr) := date; 
cache-ref(addr) :=.ref; 
ceche-mod(addr) := true 
else | state N 


new-addr := addr; | set flags so cell will be:created 


create-pkt := cmd; 
create-fiag := true 
else | must be CLR 
let crnd = CLR(addr); | 
if in-cache(addr) | state P, Ror T 
if cache-stete(addr) = "R" then 


cache-state(addr) := "R’ ” 
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sise if cache-stetetaddr) <P” then 
cache-stetetedds)::« “P’ * 
else | state T 
XMTPKT(RESO) := DONEtaddr) 
eee | state-n a 
XMTPKT(RESO) » CONE fade) 


| ++++++ end of CHOI processing +++0++ 


else 
m := true | packet wee from MEME 


else 
m = trues | mamotieg: wae: oft, must Handle: MEME inpot 


if m then 
| +++++ process packet from MEME +++ 


item := RCVPKT(MEMI) 
if item = LOAD'*\~-,--,--,--) then 
let item = LOAD'*\addr, date, ret, tag) 
if cache-state(eddr) = "P” then | know It je in conte: 
cache-dateleddr) := data; 
cache-ref(addr) := ref; 
XMTPRT(RESG): so ioe: 
if memofiag then =| can send packet at MEMO? 
memofiag = fang; = | yun 
send CLA\addr) on port MEMO; 
cache-stete(eddr) 1 “R” 
else 
cache-state(addr) := °Q° | no 
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else if cache-state(addr) = "P” then 

cache-data(addr) := dats; 

cache-ref(addr) := ref; 

XMTPKT(RESO) := item; 

if memofiag then | can send packet at MEMO? 
memotieg = false; =| yes 
send CLR(addr) on port MEMO; 
cache-state(addr) := “R’" 

else 
cache-state(addr) := "Q’ " | no 


else | must be state Q, Q', R, or RF’ 
if item = LOAD*(--,--,--,--) then | update ref and send LOAD 
cache-ref(addr) := cache-ref(eddr) + 1; 
cache-mod(addr) := true; 
XMTPKT(RESO) := LOAD*(addr, data, cache-ref(addr), tag) 
else if item = LOAD™(--,--,--,--) then 
cache-ref(addr) := cache-ref(addr) - 1; 
cache-mod(addr) := true; 
XMTPKT(RESO) := LOAD (addr, data, cache-ref(addr), tag) 
else 
XMTPKT(RESO) := LOAD\addr, data, cache-ref(addr), tag) 
else | must be DONE 
let item = DONE(addr); 
if cache-state(addr) = "R" then —_| know it is in cache 
cache-state(addr) := “T" 
else | must be state "R’” 
cache-state(addr) := "T°; 
XMTPKT(RESO) := DONE(eddr); 


| +++++4 end of MEMI processing ++++++ 
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goto A 
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APPENDIX III A 
The insertion algorithm for the rotating memory. 


flag = false | becomes true if TL already hes UPD packet for this eddress 


P = €a(X)> — | scan pointer = hash address initially 
IE RP = TLP end P= RP and —| hash addr = stert of removel region? 
(€a00> = Ca(TL(TLP))> or aX) <-afTLETLP))) then 


TL(P) := X; |.neert tem et P 
RP = RP +l mod M | shorten the removal region 
pop == pop + 1 | update TL population 
else 
if RP = TLP end P « (RP, TLP) then | hash address in removal region 


P se TLP | advance to end of remeval region 
| repeat until find empty cell or enter removal region 


until (P = RP end RP = TLP) or TLP(P)=empty or fing =true do 
( | 7 7 


| see if TL already has UPD with seme CCD address 


if aX) = a{TL(P)) and TL(P) = UPDX--,--,--) then 


flag = 1; 
else 
if (Ca(TLEP))> = €a(X)> and afX).< a{TLIP))) 
or €a(X)> €[ CaTLIP)>,P] | ts X:"ematler” then the current item? 
then : e 


Y = TL(P) | save item from TL 
TLIP) =X | insert X here 
X= Yj | incert saved item in next cell 
(which pushes everything past here) 
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P:=P+1modM | advance P to next coll 
) 


| tind out whether to insert X or process it directly 
if not flag then | insert it 

if P = RP and RP « TLP | enteced renpeat-vegion? — 

then 7 
TL(P) z= X; - {mart ete 
RP = RP + 1 mod M | shorten the remevel region 


TLEP) = XK; | insert item at P 
pop := pop+l | updete Ti. population 


ese | precees it directly 
let TL(P) = UPD\eddr, data, ref) 
it X = UPD(—-,-+-~) then Le Bi 
TLEP) := X | another UPD, new ene replaces 
else if X = FET(—~—)them = =e 
fet X = FET{--tegh =| FET, gat the dete 
XMTPKT(RESO) » LOAD(addr, deta, ref, tag) 
else if X = FET*(--,--) then 
let X = FET*(--teg, | FET*, get the dete and update ref 
TL(P) := UPDiaddr, deta, ref+1) 
XMTPKT(RESO) := LOAD (addr, deta, ref+1, tag) 
else if X = FET(--,--) then 
let X = FET (tag | FET”, get the dete and: upciete ref 
TLEP) :=:\POhedde, date, rat-1); 
XMTPKT(RESO) := LOAD (addr, deta, ref-1, tag) 
else | must be CLR - 
XMTPKT(RESO) := DONE(addr) 
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APPENDIX II! 6 
The rotating memory algorithm. 


process starts at A 
input ports CMDI, CCD! 

output ports RESO, CCDO 

var P, X, Z, eddr, data, ref, tag, CCD-addr, pop init 0, TL-cmd, 
CCD-data, CCD-ref, CCD-newref, TLP, RP 
ives | as a 


A: if TL(TLP) = empty then 
; RP = TEP; | destroy the removal region 
while TL(TLP) = empty and TLP = €CCD-eddr> do 
ea ies 
TUP = TLP +1 mod Ms | advance until catch up to CCD-sddr 
RP := TLP | keep removal region destroyed 


| look for input pechets 


it pop 2M-1 

then | TL nearly full, can’t take packets at CDI 
Z := RCVPKT(CCDI | wait for and sccept packat st CCDI 
let Z = ADDAICCD~addr, CCO-data, CCD-refh 


CCD-newref := CCD-ref 

else | can. accept packet on either port 
wait for packet at CMDI or om P = thet port | nondeterminate! 
it P = ‘COP then . 


Z = RCVPKTICCOD; =| een at CDI 
let Z = ADDR(CCO-addr, CCD-data, CCD-ref) 
CCO-newref := CCD-ret 

else 
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X := RCVPKT(CMDI); | take packet at CMD! 
|] FFF EE ete t eee t eee e eee eeeees 

| + insert or otherwise dispose of .X 

| + (from appendix III A ) 


| $4444444+0044404+440004004044 


| perform all transactions matching CCO-addr 


( 


TL-omd := TL(TLP) | remove transaction from list 

TLITLP) := empty; 

pop i= pop-1; | updete TL population mn 

RP = Ca(TL-cmd)>; __  ahartan camovel region appropriately - 

TIP TP +lodMm shies aibaleat, Yelle 

if TL-cmd = CLIKCCD-eddr) then 
XMTPKTORESO) ee DONECOCD-eddr) 


else if TL-cmd © FET(--,--) then 
jet TL-cmd = FET(addr, tag); 
XMTPKT(RESO) 2» LOAD(eddr, CCD-date, OCD-newret, tag) 


else if TL-cmd = FET*(--,--) then 
lot TL-cand » FET “(edide, tegh 


oles if TL-cmd = FET (--,--) then. 
tet TL-cmd 7. FET (addr, tegh 


CCD-newref = CCD-newret - ly 
XMTPKT(RESO) al LOAD" (addr, OCD-date, OCD-newref, tag) 
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else | must be UPD 
let TL-cmd = UPD(addr, data, ref); 
XMTPKT(CCDO) := WRITE(addr, data, ref) 


| rewrite reference count if it has changed 


if CCO-ref # CCD-newref then 
XMTPKT(CCDO) := WRITE(CCD-addr, CCD-data, CCD-newref); 


goto A 
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